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Abstract 

We  define  awareness  as  the  cognitive  interface  between  decision-makers  and  a  complex 
system,  expressed  in  a  range  of  terms  or  features,  or  specific  vocabulary  or  lexicon,  to 
describe  the  attributes  and  surrounding  environment  of  the  system.  Lexical  Link  Analysis 
(LLA)  is  a  form  of  text  mining  in  which  word  meanings  represented  in  lexical  terms  (e.g.,  word 
pairs)  can  be  represented  as  if  they  are  in  a  community  of  a  word  network.  In  the  past,  we 
have  explored  how  LLA  systematically  and  automatically  discovers  new  patterns  that  were 
previously  unknown,  and  identifies  data  dependencies  from  large-scale  defense  acquisition 
data  of  multiple  programs  that  might  be  indicators  for  program  or  investment  performances  in 
defense  acquisition  decision-making  and  research  communities. 

We  also  started  to  apply  LLA  to  improve  our  understanding  of  the  quality  of  the  data  by 
comparing  categories  of  information  and  by  detecting  data  overlaps,  inconsistency,  and  gaps 
from  a  single  program  point  of  view.  The  Acquisition  Visibility  Portal  (AVP)  is  a  critical  tool 
that  provides  the  DoD-wide  acquisition  community  with  authoritative  and  accurate  data 
services  via  interfaces  to  Defense  Technical  Center  (DTIC)  and  Defense  Acquisition 
Management  Information  Retrieval  (DAMIR)  for  programs  (e.g.,  major  defense  acquisition 
programs  [MDAPs],  acquisition  category  II  [ACATII]  programs)  with  milestones,  costs, 
schedules  and  performance  data,  selected  acquisition  reports  (SAR),  acquisition  strategy 
reports  (ASR),  the  systems  engineering  plans  (SEP),  the  test  &  evaluation  master  plans 
(TEMP),  and  the  defense  acquisition  executive  summary  (DAES),  among  others. 

The  major  advantage  of  using  LLA  is  to  apply  automation  to  reveal  and  depict — to  decision¬ 
makers — the  correlations,  associations,  and  program  gaps  across  all  the  programs  in  the 
AVP  over  many  years.  This  enables  strategic  understanding  of  data  gaps  and  potential 
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trends,  and  can  inform  managers  what  areas  might  be  highly  risky  for  a  program  and  how 
resource  and  big  data  management  might  affect  the  desired  return  on  investment  (ROI) 
among  projects. 

Introduction 

It  is  critical  that  the  Department  of  Defense  (DoD)-wide  acquisition  community  be 
able  to  access  authoritative  and  accurate  data  services  for  decision-making.  The  Acquisition 
Visibility  Portal  (AVP)  was  such  a  data  service  that  achieved  this  purpose  by  interfacing  with 
program  elements  and  warfighter  requirements  via  a  Defense  Technical  Information  Center 
(DTIC)  interface  (program  elements  [PEs],  see  [http://www.dtic.miI/descriptivesum/1;  and 
requirements,  see  [http://www.dtic.miI/doctrine/1).  The  AVP  also  included  an  interface  to 
Defense  Acquisition  Management  Information  Retrieval  (DAMIR; 

http://www.acq.osd.mil/damir/)  to  access  large,  detailed  collections  of  information  such  as 
milestones,  costs,  schedules,  and  performance  data  of  selected  acquisition  reports  (SAR) 
and  acquisition  Strategy  Reports  (ASR),  among  others,  regarding  Major  Defense  Acquisition 
Programs  (MDAPs)  and  Acquisition  Category  II  (ACATII)  program  data.  The  AVP  also 
provided  automatic  methodologies  to  systematically  improve  understanding  of  the  quality  of 
the  data. 

In  the  past,  we  have  explored  an  analytic  and  visualization  tool  named  Lexical  Link 
Analysis  (LLA),  which  we  applied  to  various  areas  of  acquisition  research,  for  example,  to 
link  warfighter  requirements  with  the  acquisition  programs  and  program  elements  (Gallup, 
MacKinnon,  Zhao,  Robey,  &  Odell,  2009;  Zhao,  Gallup,  &  MacKinnon,  2010,  2011a,  2011b, 
2011c,  201  Id,  2012a,  2012b,  2013;  Zhao,  Brutzman,  &  MacKinnon  2013).  Recently,  we 
have  started  to  explore  how  LLA  can  help  detect  data  quality,  inconsistency,  gaps,  or  bad 
data  among  categories  of  data  by  automatically  discovering  new  patterns  that  were 
previously  unknown  and  by  identifying  data  dependencies  that  might  be  indicators  for 
program  or  investment  performances. 

For  example,  we  observed  that  very  little  of  the  information  generated  for  program 
oversight  is  amenable  to  effective  analysis.  Every  major  acquisition  program’s  milestone 
review  generates  volumes  of  information,  which  the  Office  of  the  Secretary  of  Defense 
(OSD)  staff  is  supposed  to  review  to  determine  if  the  program  is  properly  prepared  for  the 
next  milestone.  Although  acquisition  professionals  and  decision-makers  at  OSD  are 
beginning  to  compile  these  artifacts  centrally  to  facilitate  review  and  analysis,  at  present  the 
only  way  to  analyze  the  information  in  these  artifacts  is  to  read  them.  With  limitations  on 
staffing,  little  time  is  available  to  thoroughly  review  these  artifacts.  Moreover,  each  functional 
community  is  required  to  review  only  the  particular  document  for  which  it  is  responsible.  For 
example,  the  systems  engineering  community  typically  only  examines  the  systems 
engineering  plans  (SEP),  the  test  and  evaluation  community  looks  only  at  the  test  & 
evaluation  master  plans  (TEMP),  and  the  acquisition  community  looks  at  the  acquisition 
strategy  reports  (ASR).  Rarely  do  any  of  these  stakeholders  review  multiple  reports  or  jointly 
discuss  them  to  determine  if  they  are  mutually  consistent  and  consider  inconsistencies  that 
might  indicate  programmatic  risk.  There  is  even  less  incentive  and  opportunity  to  look  for 
external  factors  that  would  potentially  invalidate  the  assumptions  that  underpin  the  basic 
cost,  schedule,  and  performance  targets  of  each  program’s  execution. 

These  milestone  documents,  though  distinct,  should  be  mutually  consistent.  For 
example,  they  all  describe  the  same  program,  so  should  use  consistent  naming  conventions 
and  concepts.  Divergent  naming  conventions  may  indicate  that  the  documents  were 
developed  in  isolation.  This  would  be  a  concern  because  there  are  meaningful  linkages 
between  these  reports  such  that  a  capability  defined  in  the  acquisition  strategy,  should  be 
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elaborated  in  the  systems  engineering  plan;  the  testing  of  which  should  be  described  in  the 
TEMP.  Inconsistencies  among  these  documents  may  reflect  a  risk  to  the  program. 

Motivated  by  these  situations,  we  applied  LLA  as  one  of  automatic  tools  to  examine 
large  collections  of  artifacts  for  many  programs  in  various  categories  across  the  acquisition 
and  engineering  communities.  By  using  LLA,  one  can  learn  from  the  actual  data  to  see  how 
the  common  concepts  are  expressed  in  different  artifacts  and  communities.  Overlaying  the 
concepts  for  each  category  of  artifacts  to  conduct  a  pair-wise  comparison  exposes 
significant  disconnections  between  them.  The  automatic  discovery  of  the  disconnection  or 
gaps  could  be  fed  back  to  the  human  analysts  or  decision-makers  to  perform  further 
investigations. 

In  this  paper,  we  demonstrate  a  set  of  comprehensive  LLA  analysis  reports  and 
visualizations  generated  automatically  for  a  given  program  using  multiple  categories  of 
program  data  as  data  sources.  These  reports  and  visualizations  revealed  that  there  are  data 
correlations  and  gaps  among  at  least  eight  data  sources.  These  correlations  and  gaps  could 
form  the  basis  for  further  inquiry  or  future  reconciliation  of  the  expectations  (e.g.,  acquisition 
strategy)  and  realities  (e.g.,  engineering  feasibility)  from  various  communities  for  a  same 
MDAP  program.  LLA  is  able  to  discover  in  detail  where  the  gaps  and  inconsistencies  of  the 
data  across  multiple  data  sources,  which  lead  to  identify  and  offer  specific  and  productive 
directions  for  further  examination  regarding  why  gaps  occur  and  where  they  exist. 

Methodology 

Detecting  Data  Consensus  and  Gaps  in  the  AVP  Data  Sources 

To  detect  the  data  gaps  between  two  categories  of  information,  LLA  compares  the 
milestone  artifacts  for  a  given  program  from  one  category  to  another,  for  example, 
comparing  the  ASRs  with  the  SEPs  for  program  “X”  at  Milestone  B.  These  comparisons, 
reported  as  themes,  features,  and  word  pairs,  may  help  cue  a  decision-maker’s  attention  to 
the  potential  issues  and  consider  specific  and  productive  directions  for  further  scrutiny. 

To  illustrate  the  methodology,  we  first  manually  extracted  a  sample  Navy  ship¬ 
building  program  as  a  representative  of  Major  Defense  Acquisition  Programs  (MDAPs)  from 
the  AVP  with  sources  including  various  artifacts,  for  example,  Acquisition  Strategy  Reports 
(ASR),  Selected  Acquisition  Reports  (SAR),  Defense  Acquisition  Executive  Summaries 
(DAES),  Certification  Milestone  B,  Acquisition  Decision  Memorandums  (ADM),  Acquisition 
Program  Baselines  (APB),  Technology  Readiness  Assessment  (TRA),  Test  &  Evaluation 
Master  Plans  (TEMP),  and  Systems  Engineering  Plans  (SEP). 

When  using  LLA  to  compare  two  data  sources,  for  example,  comparing  ASR  and 
TEMP,  we  asked  the  question,  “What  are  the  features  or  clusters  of  features  (e.g.,  themes) 
discussed  in  ASR  but  not  discussed  in  TEMP?”  If  found,  there  might  be  various  reasons  to 
explain  the  discrepancies.  For  example,  if  features  appear  only  in  ASR  but  not  in  TEMP,  it 
could  be  a  gap  because  of  (1 )  a  data  quality  issue  (e.g.,  a  mishandling  of  data  by  AVP),  (2) 
a  data  classification  issue  (e.g.,  unclassified  data  vs.  classified  data),  or  (3)  a  real 
requirement  gap  (i.e.,  a  concept  required  by  acquisition  for  which  no  engineering  feasibility 
document  or  blueprint  can  be  located).  These  types  of  information,  if  detected  earlier,  could 
provide  decision-makers  with  the  basis  to  make  earlier  amendments,  thereby  reducing 
program  risks  and  future  costs. 

It  is  also  noted  that  some  inconsistencies  might  be  related  to  the  distinctive  functional 
perspective  of  each  report.  For  example,  one  would  expect  a  higher  occurrence  of  concepts 
related  to  acquisition  strategy:  that  is,  contracts,  budget,  funding,  competition.,  etc.,  whereas 
a  Systems  Engineering  Plan  would  have  a  higher  occurrence  of  concepts  related  to 
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architecture,  systems,  integration,  interface,  specifications,  and  so  forth.  A  lack  of 
consistency  of  these  functionally-specific  terms  from  one  artifact  to  another  would  not 
necessarily  represent  increased  risk. 

In  this  report,  we  report  on  the  overall  comparisons  of  these  data  sources.  The 
FOUO  content  is  not  included  in  the  main  body  of  the  report.  These  questions  can  be 
answered  by  human  analysts  if  they  have  enough  time  to  go  through  the  piles  of  artifacts 
one  by  one  and  mark  the  differences.  The  advantage  of  using  LLA  is  to  automate  the 
process  and  thereby  provide  indepth  initial  screening  prior  to  human  analysis. 

Lexical  Link  Analysis  (LLA)  Method 

A  review  of  LLA  can  be  found  in  Appendix  B  to  provide  a  basis  for  how  this  analysis 
is  accomplished. 

Research  Results 
Data  Access 

This  year  we  experienced  improved  data  access  to  the  following  portals: 

•  Acquisition  Visibility/DMAIR  Portal 
(https://ebiz.acq.osd.mil/DAMIR/PortalMain/DamirPortal.aspx) 

•  Acquisition  Visibility/AI R  Portal 

(https://www.dodtechipedia.mil/dodc/pluqins/AIR/airdocuments.action) 

•  Acquipedia  (https://dap.dau.mil/acquipedia).  A  screenshot  of  this  data  source 
is  shown  in  Figure  1. 
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Initial  Capabilities  Document  (ICO) 

The  Inhel  Cecebtkies  Document  (ICO)  document*  the  OcO  need  for  e  material  aco'oach  (or 
an  approach  that  coiierei  malarial  and  norvmetenel  solution  eat*)  to... 

Analysis  of  Alternatives  (AoA) 

The  AoA  muum  potential  materiel  col ul one  to  utwfy  the  capebiMy  need  documented  m  the 
approved  Impel  Caoabrloe*  Document  (ICO).  It  focuses  on  idem... 

Copebility  Development  Document  (ODD) 

A  document  that  captures  the  information  necessary  to  develop  e  proposed  program^), 
normally  u«ng  an  cvokSionery  scorns*  ion  (EA)  strategy-  The  COO  outlines  ... 

Acquisition  Category  (ACAt) 

Categories  cstatAshcd  lo  feaUstc  decentralized  decision  malng  and  execution  and 
oompkanoe  wth  statutorily  imposed  requirements.  The  categories  determine  ... 

Test  ft  Evaluation  Master  Plan  (TEMP) 

OAU  Glossary:  The  T**tP  documents  the  over  at  Mruct 
Cvaljabcn  (T&£)  program.  It  provides  a  framework  w 

Net  Ready  Key  Perlomiance  Parameter  (KPP) 

Documents  sponsor  identified  and  >ont  Requirement  Oversight  Council  (JROC)  vaidated 


IXIlmlPfl1  Analysis 

OOTmlPr  9  If  the  DoD  acronym  that  partem*  to  th 


Acquisition  Program  Raswllna  (APB) 

From  the  OAU  Glossary  BawAne  that  reflect*  th 

movnum  number  of  cost,  schedule,  end  performance  attributes  Chat  desc... 
Concept  ol  Operations  (CONOPS) 

A  verbal  or  graphic  ttatemerc  that  dearly  and  concisely  expresses  what  the  j 
commander  r«endj  to  accompli**  and  how  it  mil  be  done  using  avaiab'e  ... 


Operations  ft  Maintenance  Funds 

Operations  and  Marcenance  (OAM)  funds  include  many  separate  appropriate*!*  (e.g.. 
Operation*  and  MarSenence  Army,  Opcratione  and  Maintenance  Navy,  Operation*  ... 
Rr-.r-.ir,  h  DryHopnirnt  Inst  H  I  valuation  (RDTftt  )  1  unds 


c  Q 

3AU 

I&nisj  i±i 


aivioil  toiies 


B2*,'2013  100.0M 


I12i20C0  9S.02S 


IS30IJ  ST.aeo 


BA2013  S3  317 


1/24/201 1  02.S20 


A'2S.'20I1  01.7: 


9/1W20II  77.071 


Figure  1.  Acquipedia  Data  Source 
Automatic  Data  Streaming 

Our  automated  data  analysis  efforts  are  still  in  the  process  of  development.  Although 
as  individuals,  the  researchers  in  this  project  are  able  to  access  some  individual  programs  in 
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the  AVP  following  the  proper  request  processes,  large-scale  streaming  these  data  to  a  tool 
residing  outside  AVP  proves  difficult  due  to  policy  issues.  For  example,  according  to  the  AIR 
help  desk,  “Access  to  the  documents  in  AIR  is  controlled  by  each  individual  who  uploads  the 
documents;  therefore,  if  one  wants  access  to  a  specific  document  that  you  do  not  currently 
have  access  to,  one  has  to  select  the  little  envelope  icon  (in  the  search  results  page)  to 
send  an  email  to  the  document  uploader,  requesting  access  to  that  document.  It  is  up  to  that 
person  to  grant  or  deny  access  to  that  document.” 

Therefore,  we  focused  on  following  the  current  DAMIR/AIR  data  access  policies  by 
sending  requests  program  by  program  and  file  by  file  to  get  samples  of  AVP.  If  the  LLA 
results  are  proven  useful  to  the  analysts  and  decision  makers,  the  LLA  tool  can  be  installed 
inside  the  AVP  so  that  it  is  close  to  the  data  rather  than  to  request  access  then  download 
each  data  set. 

We  tested  the  process  for  one  ship-building  program.  Data  was  downloaded  with 
documents  from  the  following  nine  sources: 

1 .  TEMP:  Test  &  Evaluation  Master  Plan 

2.  SEP:  Systems  Engineering  Plan 

3.  ASR:  Acquisition  Strategy  Report 

4.  SAR:  Selected  Acquisition  Report 

5.  DAES:  Defense  Acquisition  Executive  Summary 

6.  ADM:  Milestone  B  2366b  Certification  Acquisition  Decision  Memorandum 

7.  APB:  Acquisition  Program  Baseline 

8.  TRA:  Technology  Readiness  Assessment 

9.  LCSP:  Life  Cycle  Sustainment  Plan 

Many  of  these  documents  are  for  official  use  only,  and  the  SEP  documents  for  the 
ship-building  program  studied  in  this  paper  are  still  in  progress  to  be  extracted  at  the  time  of 
this  paper.  Therefore,  we  only  include  eight  data  sources  in  the  following  discussions. 

Conversely,  we  were  able  to  successfully — and  automatically — download  publically 
accessible  Acquipedia  data  into  the  LLA  tool  using  the  crawler  technology  including  380 
html  pages,  which  serves  as  an  online  encyclopedia  of  common  defense  acquisition  topics. 
Each  topic  is  identified  as  an  article;  each  article  contains  a  definition,  a  brief  narrative  that 
provides  context,  and  links  to  the  most  pertinent  policy,  guidance,  tools,  practices,  and 
training  that  further  augment  understanding  and  expand  depth.  Since  it  contains  standard 
terminologies  for  common  defense  acquisition  topics,  we  are  in  the  process  of  using  the 
data  as  a  separate  data  source  for  supervised  learning  data  to  train  LLA  to  improve  the 
understanding  of  context-dependent  meaning. 

Automatic  Generation  of  Reports  and  Visualizations  Showing  the  Relations  of 
the  Data  Sources 

For  the  one  ship-building  program  we  have  downloaded  the  eight  data  sources  from 
AVP  (DAMIR  and  AIR)  as  we  explained,  our  objectives  were  to  compare  the  eight  data 
sources  to  see  similarities  and  differences.  In  order  to  answer  the  question  such  as  “What 
are  the  features  or  clusters  of  features  (e.g.,  themes)  discussed  in  ASR  but  not  discussed  in 
LCSP?”  for  every  pair  of  data  sources  in  comparison,  we  focused  on  the  automatic 
generations  of  the  following  reports  given  the  data  for  each  source. 
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1.  Match  Matrix  Report: 

Table  1  shows  match  scores,  i.e.,  number  of  word  pairs  matched  among  any 
of  two  data  sources  in  the  eight  data  sources  and  correlations  among  these 
categories.  A  match  score  for  a  data  source  is  the  total  number  of  matched 
features  (e.g.,  LLA  word  pairs)  and  a  uniqueness  score  is  the  total  number  of 
unique  word  pairs  are  unique  to  the  source.  A  correlation  is  normalized  using 
the  match  score  and  uniqueness  score.  For  example,  a  correlation  0.15 
between  ASR  and  Acquipedia  is  computed  as  =  499/(^(943  +  3944))  x 
V( 832  +  1415)). 

Table  1.  Match  Scores  and  Correlations  for  the  Eight  Data  Sources 


Match  Score 

Acquipedia 

ASR 

LCSP 

SEP 

DAES 

TEMP 

ADM 

SAR 

APB 

TRA 

Uniqueness  Score 

1 

Acquipedia 

943.00 

499  00(0  151 

521.00(0.221 

292.00(0.191 

86.00(0.051 

82  00(0  061 

25.00(0.041 

51.00(0.091 

3.00(0.011 

5  0010  Oil 

3944.00 

2  1 

ASR 

832.00 

499  00(0.15') 

251  00(0  151 

194.00(0.181 

97.00(0.091 

78  00(0  081 

18.00(0.051 

30.00(0.081 

5.00(0.031 

4  0010  011| 

1415.00 

3 

LCSP 

513.00 

521  00(0.22') 

251  00(0  151 

119.00(0  161 

45  00(0.061 

34  00(0  051 

6.00(0.021 

15.00(0  051 

2.00(0.021 

3  ooro  onl 

657.00 

4 

SEP 

239.00 

292.00(0.191 

194.00(0.181 

119.00(0.161 

34.00(0.071 

1 1  00(0  021 

6.00(0.031 

10.00(0.061 

0.00(0.001 

3  oom  021 

253.00 

5 

DAES 

175.00 

86.00(0.051 

97.00(0.091 

45.00(0.061 

34.00(0.071 

11.00(0.021 

6.00(0.031 

25.00(0.131 

1.00(0.011 

l.ooro.onl 

368.00 

6 

TEMP 

86.00 

82.00(0.061 

78.00(0.081 

34.00(0.051 

11.00(0.021 

11.00(0.021 

0.00(0.001 

2.00(0.011 

0.00(0.001 

o  ooro.ooil 

353.00 

7 

ADM 

46.00 

25.00(0.041 

18.00(0.051 

6.00(0.021 

6.00(0.031 

6.00(0.031 

0.00(0.001 

0.00(0.001 

0.00(0.001 

i  ooro  02i 

20.00 

8 

SAR 

40.00 

51.00(0.091 

30.00(0.081 

15.00(0.051 

10.00(0.061 

25.00(0.131 

2.00(0.011 

0.00(0.001 

0.00(0.001 

i  ooro  02i 

25.00 

9 

APB 

8.00 

3  00(0  011 

5  00(0.031 

200(0.021 

0  00(0.001 

1.00(0  011 

0  00(0  001 

0.00(0.001 

000(0  001 

o  ooro  ooii 

7.00 

10 

TRA 

8.00 

5.00(0.011 

4.00(0.011 

3.00(0.011 

3.00(0.021 

1.00(0.011 

0.00(0.001 

1.00(0.021 

1.00(0.021 

0.00(0.001 

48.00 

2.  Visualization  of  Correlations  Among  Data  Sources: 

Figure  2  (a)  shows  a  network  using  correlations  of  the  eight  data  souces  in 
Table  1.  Figure  2  (b)  shows  the  network  connections  with  the  correlations  > 
0.1  in  (a).  In  this  example,  data  sources  LCSP,  ASR  and  SEP  have  the 
highest  correlations  with  Acquipedia  and  with  each  other,  indicating  ASR, 
LCSP  and  SEP  may  use  more  standardized  vocubarlies  and  terminologies 
than  other  data  sources  and  are  also  correlated  with  each  other  much  more 
than  with  the  other  sources.  SAR  and  DAES  are  also  correlated  with  each 
other  more  with  each  other. 
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Figure  2.  (a)  A  Network  Depiction  of  Correlations  of  the  Eight  Data  Sources  in  Table 

1;  (b)  The  Correlations  >  0.1  in  (a) 
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3.  Match  Matrix  Drill-down  Report: 

Figure  3  shows  part  of  the  drill-down  capabilities  for  the  list  of  the  matched 
terms  between  Aaquipedia  and  all  other  data  sources.  The  terms  are  listed 
according  the  bi-gram  frequencies  and  probabilities.  The  hyperlinks  under 
these  terms  lead  to  a  search  and  drill-down  report  to  see  the  locations  of  the 
terms  used  in  the  documents.  The  related  concepts  can  be  also  visualized. 


Frequencies/probability  of  the  word  pairs 


[163.00.0.36]JOIXT  CAPABILITIES  (m  Theme  6"\Anomaly) 

(T59.00.0.281LIFE  CYCLE  SUSTAINMENT  (in  Theme  15 7. Popularity) 

[147  00.0.251ACQITSUIQN  PROGRAM  C»  Theme  157.PopuIarity) 
(122.00.0.44]LQG1ST1CS  SUPPORT  fin  Theme  169, Emerging) 

fl  12  00-0  061SYSTEM  MANAGEMENT  ( m  Theme  172.Popularity) 

[111 .00. 1  001PERFORMAXCE  INDEX  (in  Theme  46.Exnerging) 

[105.00.0.63)MAXAGER  PSM  (in  Theme  108Aaomaiy) 

[103-00.1. OQ)INFORMATION  ASSURANCE  On  Theme  4 6 .Emerging) 
ri02.00-0.201TEST  EVALUATION  (in  Theme  104  .Emerging) 

[102.00.0.3 ^INTEGRATED  PROGRAM  (in  Theme  2 2, Anomaly) 

[99  00.0.37]GQVERXMEXT  PROPERTY  C®  Theme  '5  Anomaly) 

[82  00.1  OOJCOST  ESTIMATING  (in  Theme  176. Popularity) 

[80  00.0.471DAIAELQHIS  C«  Theme  46,Emerging) 

[79.00.0. 18|SUPPQRT  EQUIPMENT  (in  Theme  6~  Anomaly) 

[78  00.0  82)WEAPOX  SYSTEMS  (m  Theme  172,Popularit>) 

[76.00.0. 1SJLIFE  CYCLE  COST  (in  Theme  10S.Anomah) 

[75  00.0  37]PLANNI>'0  PACKAGE  (in  Theme  1 20,Emerging) 

[73 .00.0.2  lJMAXAGEMEXT  OFFICE  (in  Theme  131Anomaly) 
[73.00.0.251MAIXTENAXCE  PLANNING  C®  Theme  120£merging) 

[72.00.0  071REQITREMEXTS  MANAGEMENT  (in  Theme  154.EnwTging) 
[70.00,0. 19)\IANAGEMENT  IMPLEMENTATION  Cm  Theme  1 1 5,Anomah) 
[68  00.0  101CQXTRACT  PERFORMANCE  (in  Theme  46.Emergmg) 

[68.00.0  14JSYSTEM  PERF QRMAXCE  On  Theme  46,Emergmg) 

[68  00.0.0S1DATA  MANAGEMENT  (in  Theme  46.Emerging) 

[68  00. 1  00)\LANUF  AC  TURING  DE\~ELOPMENT  C®  Theme  1 57.Popdarity) 
[61.00.0.13)SYSTEM  LE\* *EL  tin  Theme  106.Anomalv) 

[61  00.0  621PROGRAM  MANAGERS  (in  Theme  42.Anomah) 

[60  00.0.331FEDERAL  ACQUISITION  On  Theme  1 5 4. Emerging) 

f58.00.0.20[KEY  SYSTEM  (in  Theme  99, Emerging) 

[57.00.0  24)CQMMERCIAL  ITEMS  (in  Theme  91  Anomaly) 

[56  00,1  001PRQGR.AM  EXICUTIM  (m  Theme  r6,Popularit>) 


Clickthe  linkto  search/drill  down  to  the  original  documents 


Related  concepts  visualized 


Figure  3.  Drill-Down  Capabilities  for  Data  Source  Acquipedia 

4.  Discovered  Theme  Report: 

Table  2  shows  the  list  of  themes  (clusters  of  word  pairs)  when  comparing  two 
data  sources,  e.g.,  ASR  and  Acquipedia.  We  report  here  the  following 
statistics  in  Theme  157(P),  176(P),  and  so  on. 

•  Number  of  unique  features  for  Source  1  (e.g.,  ASR) 

•  Number  of  unique  features  for  Source  2  (e.g.,  Acquipedia) 

•  Matched  features  for  both  sources 

•  Correlation  of  two  sources  (or  consensus  rate),  i.e.,  percentage  of  the 
features  that  are  matched 

•  Gap  rate:  percentage  of  the  features  that  are  not  matched 

These  statistics  show  in  which  areas  (reflected  in  the  themes)  the  two  data  sources 
agree  (consensus,  e.g.,  46[E])  and  disagree  (gap, e.g.,  167[P])  the  most.  Clicking  on  the 
Visualization  column  of  46(E)  and  167(P)  lead  to  the  visualizations  of  two  areas  where 
consensus  and  gap  took  place. 
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Table  2. 


List  of  Themes  Comparing  Two  Data  Sources  (e.g.,  ASR  and  Acquipedia) 


Theme  Id 

=  of  Unique  Features  for  Source  ASR 

#  of  Unique  Features  for  Source  Acquipedia 

=  of  Matched  Features 

Consensus  Rate(Correlation) 

Gap  Rate 

Visualization 

157(P) 

23 

206 

34 

0.13 

0.87 

157(P) 

176(P) 

42 

138 

47 

0.21 

0.79 

176TPY 

16?(P) 

23 

222 

16 

0.06 

0.94 

167(P) 

172(P) 

38 

94 

23 

0.15 

0.85 

172(P) 

169(E) 

27 

119 

31 

0.18 

0.82 

169(E) 

46(E) 

15 

117 

46 

0.26 

0.74 

46(E) 

104(E) 

26 

64 

12 

0.12 

0.88 

104(E) 

120(E) 

30 

106 

27 

0.17 

0.83 

120(E) 

111(E) 

14 

73 

8 

0.08 

0.92 

111(E) 

154(E) 

11 

94 

21 

0.17 

0.83 

154(E) 

99(E) 

22 

86 

27 

0.20 

0.80 

99(E) 

141(E) 

19 

65 

7 

0.08 

0.92 

141(E) 

179(A) 

29 

87 

27 

0.19 

0.81 

179(A) 

124(A) 

17 

54 

10 

0.12 

0.88 

124(A) 

106(A) 

28 

72 

16 

0.14 

0.86 

106(A) 

67(A) 

20 

59 

23 

0.23 

0.77 

67(A) 

108(A) 

20 

86 

22 

0.17 

0.83 

108(A) 

32(A) 

11 

61 

6 

0.08 

0.92 

32(A) 

149(A) 

10 

81 

11 

0.11 

0.89 

149(A) 

22(A) 

21 

68 

8 

0.08 

0.92 

22(A) 

166(A) 

16 

81 

10 

0.09 

0.91 

166(A) 

131(A) 

7 

80 

6 

0.06 

0.94 

131(A) 

115(A) 

12 

106 

3 

0.02 

0.98 

115(A) 

66(A) 

10 

59 

4 

0.05 

0.95 

66(A) 

68(A) 

16 

58 

16 

0.18 

0.82 

68(A) 

145(A) 

23 

70 

7 

0.07 

0.93 

145(A) 

5.  Discovered  Consensus  and  Gap  Visualization  Among  Data  Sources: 

Figure  4  shows  a  visualization  of  comparison  of  two  sources,  e.g.,  the  data 
source  LCSP  and  ASR  for  the  same  ship-building  program  in  an  effort  to 
discover  the  consensus  and  gaps  among  the  multiple  data  sources  for  the 
same  program.  Red  nodes  show  the  most  “central”  nodes,  used  as  keywords 
to  summarize  this  theme,  i.e.,  “Program,  Cost,  Test,  Additional.”  Red  Links 
show  word  pairs  shared  by  the  two  sources.  Yellow  Links  show  the  unique 
word  pairs  from  one  source  (e.g.,  LCSP),  and  green  links  show  word  pairs 
from  the  other  source  (e.g.,  ASR).  The  actual  word  pairs  are  eliminated  here 
since  the  content  is  for  official  use  only.  The  consensus  rate  for  this  theme  is 
29%,  i.e.,  29%  of  word  pairs  or  features  are  in  agreement,  71%  of  word  pairs 
are  not.  As  one  can  see,  ASR  focuses  on  “Test”  and  LCSP  does  not. 
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Figure  4.  Visualizations  of  Comparison  of  Two  Sources 

6.  Discovered  Consensus  and  Gap  Report  among  Data  Sources: 

Figure  5  shows  a  report  listing  of  word  pairs  matched  and  unique  in  the  two 
data  sources  (e.g.,  LCSP  and  ASR)  for  the  theme  in  Figure  4.  The  actual 
word  pairs  are  eliminated  here  since  the  content  is  for  official  use  only. 
Numbers  in  the  brackets  are  the  bi-gram  frequencies  and  probabilites 
respectively. 
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•  Red  nodes:  the  most  'cent raT  nodes,  used  as  keywords  to  summarize  ttw  theme 

•  Red  Lmks:  word  pairs  shared  by  at  least  two  sources 

»  Green  Units:  word  pars  from  Source 

Total  Consensus  Features  for  ICSP.ASR:  26 
Total  Umquness  Features  for  LCSP:  14 

Total  Umquness  Features  for  ASR:  49 

Gap  Rate:  0.71 

Consensus  Rate:  0.29 

Unique  Features  for  LCSP 

unique  Features  for  ASR 

[11 

[2] 

[23- 

[2] 

[33 
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[21 
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[21 

[12 

[2! 

[21 

[3] 

[23 

[237 

[21 

[6] 

[41 

[6fi 

[23 

[43  I 

[30 

[61 

[6] 

[33 

[23 

[3] 

[73 

[21 

[31 

[10 

[21  ) 

[21 

[63 

[33 

[31 

[73 

[21 

[23 

[41 

CS] 

[21 

[15 

[21 

[2] 

[®] 

[33 

[21 

[23 

[33 

ro 

[4J 

[23 

[23 

nr 

[21 

[13 

[21 

[17.  .  . 

[33 

[2] 

[2] 

[2] 

[31 

[2] 

[4] 

[2] 

[2] 

[21 

[21 

[33 

[2] 

[3] 

Figure  5.  List  of  Word  Pairs  Matched  and  Unique  in  Two  Data  Sources 
Acquipedia 

LLA  generated  910  word  pairs  (out  of  5,062  total  word  pairs)  from  Acquipedia  that 
are  matched  from  this  program’s  seven  data  sources.  The  complete  matched  list  can  be 
found  in  Appendix  A,  which  can  be  considered  as  a  set  of  standardized  vocabularies  and 
terminologies  used  by  this  ship-building  program.  Among  the  910  matched  features,  39% 
are  anomalous  features,  35%  are  emerging  features  and  25%  are  popular  features. 


Table  3.  Breakdown  of  Matched  Features 


Percentage 

Number  of 
Features 

Anomalous  features 

39% 

358 

Emerging  features 

35% 

322 

Popular  features 

25% 

230 

The  outputs  of  LLA  are  then  divided  into  three  types: 

•  Popular  or  Normal  (P):  themes  containing  the  highest  number  of  mutually 
connected  word  pairs.  The  themes  represent  the  main  topics  in  a  corpus  at 
the  time.  They  may  be  also  regarded  as  less  interesting  because  they  are 
already  in  the  public  consensus  and  awareness,  therefore,  less  room  for 
growth. 

•  Emerging  (E):  themes  containing  the  medium  number  of  mutually  connected 
word  pairs,  these  themes  may  grow  to  popular  over  time  as  we  show  later  in 
the  examples. 
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•  Anomalous  (A):  themes  containing  the  lowest  number  of  mutually  connected 
word  pairs.  These  themes  may  be  off-topics  which  may  seem  they  do  not 
belong  here  compared  to  other  ones  and  may  be  interesting  for  further 
investigation. 

This  validates  that  our  previous  observations  that  anomalous  and  emerging  features 
are  more  interesting  because  they  are  used  in  the  documents  regarding  an  actual  ship¬ 
building  program. 

Planned  Future  Work 

Future  investigations  are  planned  as  the  following  additional  studies: 


•  Continue  working  with  Sponsors  and  AVP  analysts  to  develop  a  process  to 
generate  the  LLA  reports  and  visualizations  for  any  given  program  in  AVP. 

•  Study  the  program  interactions  for  a  portfolio  of  programs.  For  example,  one 
of  the  biggest  risk  factors  in  defense  acquisition  is  the  unanticipated  effects  of 
program  interactions.  Some  current  work  exists  toward  identifying 
interdependence  among  programs  within  a  system  of  systems  (SoS; 
Dahmann  et  al. ,  2005).  Yet,  more  broadly,  and  as  a  result  of  required  joint 
capabilities,  portfolios  often  include  program  interdependencies  and  SoS 
effects.  Ultimately,  the  current  “program-centric”  acquisition  paradigm  is 
increasingly  ill-suited  to  identify  and  address  program  risks  that  arise  outside 
of  program  boundaries.  We  will  select  a  portfolio  of  programs  and  focuses  on 
one  type  of  data  sources,  for  example  ASR  to  see  if  LLA  can  depict  the 
interaction  risks. 

•  Use  Acquipedia  as  a  separate  data  source,  which  contains  standard 
terminologies  for  common  defense  acquisition  topics,  for  supervised  learning 
data  to  train  LLA  to  improve  the  understanding  of  context-dependent 
meaning. 


Conclusion 

In  this  paper,  we  demonstrate  a  set  of  comprehensive  LLA  analysis  reports  and 
visualizations  generated  automatically  for  a  given  AVP  program  using  multiple  categories  of 
program  data  as  data  sources.  These  reports  and  visualizations  reveal  that  there  are  data 
correlations  and  gaps  among  at  least  eight  data  sources.  These  correlations  and  gaps  could 
form  the  basis  for  further  inquiry  or  future  reconciliation  of  the  expectations  (e.g.,  acquisition 
strategy)  and  realities  (e.g.,  engineering  feasibility)  from  various  communities  for  a  same 
MDAP  program.  LLA  is  able  to  discover  in  detail  where  the  gaps  and  inconsistencies  of  the 
data  across  multiple  data  sources  reside,  which  in  turn,  can  lead  to  the  identification  of 
future  specific  and  productive  directions  for  further  examination  regarding  why  gaps  occur 
and  where  they  exist. 
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Appendix  A:  Word  Pairs  in  Acquipedia  Matched  in  a  Ship-building  Program 

r682.00.0.341PRQGRAM  MANAGEMENT  (in  Theme  1 76, Popularity) 

[3 1 8.00,0.331SYSTEMS  ENGINEERING  (in  Theme  1 08, Anomaly) 
r280.00,0.921PRQGRAM  OFFICE  (in  Theme  131, Anomaly) 
f241 .00.0.601TECHNICAL  DATA  (in  Theme  46, Emerging) 

[  1 80.00, 1  .OOIPROGRAM  MANAGER  (in  Theme  1 08, Anomaly) 
ri80.00.0.241LIFE  CYCLE  MANAGEMENT  tin  Theme  108, Anomaly) 
r  173 .00.0. 131 ACQUISITION  MANAGEMENT  (in  Theme  1 57, Popularity) 
r  163 .00.0.361 JOINT  CAPABILITIES  (in  Theme  67, Anomaly) 

[160.00,0.541PRQDUCT  SUPPORT  MANAGER  (in  Theme  108, Anomaly) 
ri59.00,0.281LIFE  CYCLE  SUSTAINMENT  (in  Theme  15 7, Popularity) 
r  147.00.0.251ACQUISITIQN  PROGRAM  (in  Theme  15 7, Popularity) 

U22.00,0.441LQGISTICS  SUPPORT  (in  Theme  169, Emerging) 
ri  19.00,0.321LEVEL  MAINTENANCE  (in  Theme  106, Anomaly) 
ft  12.00,0.061SYSTEM  MANAGEMENT  (in  Theme  1 72, Popularity) 

[111 .00. 1 .001PERFORMANCE  INDEX  (in  Theme 46, Emerging) 
ri08.00,0.351RISK  MANAGEMENT  (in  Theme  176, Popularity) 
ri05.00.0.631MANAGER  PSM  (in  Theme  108, Anomaly) 
f  1 03 .00, 1  .OOIINFORMATION  ASSURANCE  (in  Theme  46, Emerging) 
ri02.00.0.201TEST  EVALUATION  (in  Theme  1 04, Emerging) 
ri 02.00.0.3 11INTEGRATED  PROGRAM  (in  Theme  22, Anomaly) 
r99.00.0.371GQVERNMENT  PROPERTY  (in  Theme  75, Anomaly) 

[82.00, 1.001COST  ESTIMATING  (in  Theme  1 76, Popularity) 
r80.00.0.471DATA  RIGHTS  (in  Theme  46, Emerging) 

[79.00,0. 1 81SUPPORT  EQUIPMENT  (in  Theme  67, Anomaly) 

[78 .00,0. 821 WEAPON  SYSTEMS  (in  Theme  172, Popularity) 

[76.00,0. 181LIFE  CYCLE  COST  (in  Theme  108, Anomaly) 
r75.00.0.371PLANNING  PACKAGE  (in  Theme  120, Emerging) 
f73 .00,0.2 11MANAGEMENT  OFFICE  (in  Theme  131, Anomaly) 
r73.00,0.251MAINTENANCE  PLANNING  (in  Theme  120, Emerging) 
r72.00.0.071REQUIREMENTS  MANAGEMENT  (in  Theme  1 54, Emerging) 
r70.00,0. 1 91MANAGEMENT  IMPLEMENTATION  (in  Theme  1 15, Anomaly) 
r68.00.0. 1 01CONTRACT  PERFORMANCE  (in  Theme  46, Emerging) 
r68.00,0.141SYSTEM  PERFORMANCE  (in  Theme  46, Emerging) 
r68.00.0.081DATA  MANAGEMENT  (in  Theme  46, Emerging) 
r68.00, 1 .001MANUFACTUR1NG  DEVELOPMENT  (in  Theme  15 7, Popularity) 
r61.00.0.131SYSTEM  LEVEL  (in  Theme  106, Anomaly) 
r61.00,0.621PRQGRAM  MANAGERS  (in  Theme  42,Anomaly) 
r60.00,0.331FEDERAL  ACQUISITION  (in  Theme  154, Emerging) 
r59.00,0.5010PERATIQNAL  CAPABILITY  (in  Theme  32, Anomaly) 
r58.00.0.201KEY  SYSTEM  (in  Theme  99, Emerging) 
r57.00,0.241CQMMERCIAL  ITEMS  (in  Theme  91, Anomaly) 
r56.00,l  .OOIPROGRAM  EXECUTIVE  (in  Theme  176, Popularity) 

[56.00,0. 16]SYSTEMDESIGN  (in  Theme  99, Emerging) 

[55.00,0.201IMPLEMENTATIQN  GUIDE  (in  Theme  1 15, Anomaly) 
r53.00.0.161SQFTWARE  SUPPORT  (in  Theme  1 69, Emerging) 

[52.00,0.071ACQUISITIQN  PROCESS  (in  Theme  15 7, Popularity) 

[49.00,0.421CQST  EFFECTIVE  (in  Theme  99, Emerging) 

[49.00,0. 161CONTRACT  BUDGET  (in  Theme  1 76, Popularity) 

[49.00,0.341LIFE  CYCLE  SUPPORT  (in  Theme  108, Anomaly) 

[48.00,0.321INTERFACE  DESIGN  (in  Theme  99, Emerging) 

[47.00,0.671PRQGRAM  BASELINE  (in  Theme  1 76,Popularity) 

[47.00, 1.001 ACQUISITION  PROGRAMS  (in  Theme  1 20, Emerging) 

[46.00,1 .001ACQUISITIQN  TECHNOLOGY  (in  Theme  15 7, Popularity) 

[46.00,0. 501DQD  COMPONENT  (in  Theme  15 7, Popularity) 

[46.00,0.041CQNTRACT  MANAGEMENT  (in  Theme  1 67,Popularity) 

[45.00,0.501SUSTAINMENT  METRICS  (in  Theme  15 7, Popularity) 

[45.00,0.351HUMAN  SYSTEMS  (in  Theme  108, Anomaly) 
r44.00.0.081SYSTEM  SUSTAINMENT  (in  Theme  15 7, Popularity) 

[43.00,0.371INDEPENDENT  LOGISTICS  (in  Theme  176, Popularity) 
r42.00.0.371INTEGRATED  TEST  (in  Theme  22, Anomaly) 

[4 1 .00,0.331CAP ABILITY  DEVELOPMENT  (in  Theme  32, Anomaly) 
r39.00,0.501DEVELQPMENTAL  TEST  (in  Theme  179, Anomaly) 

[39.00,0. 1 81INFORMATION  SYSTEM  (in  Theme  46, Emerging) 

[38.00,0.441RELIABILITY  MAINTAINABILITY  (in  Theme  153, Anomaly) 

[38.00,0.401INVENTQRY  CONTROL  (in  Theme  1 72, Popularity) 
r36.00.0.031INFQRMAT10N  SUPPORT  (in  Theme  46, Emerging) 

[36.00,0.401JQINT  INTEROPERABILITY  (in  Theme  70, Anomaly) 

[35.00,0.021ACQUISITIQN  POD  (in  Theme  15 7, Popularity) 
r35.00.0.051TECHNICAL  PERFORMANCE  (in  Theme  46, Emerging) 

[35.00,0. 171MANAGEMENT  MODEL  (in  Theme  179, Anomaly) 
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r34.00,0.051LIFE  CYCLE  SYSTEM  (in  Theme  108, Anomaly) 

[34.00,0. 1711NITIAL  CAPABILITIES  (in  Theme  106, Anomaly) 

[34.00  0.541PERFQRMANCE  SPECIFICATION  (in  Theme  172, Popularity) 
r34.00.0. 1 71DEFENSE  PROGRAM  (in  Theme  124, Anomaly) 

[33 .00,0. 1 31INTEGRATION  DEVELOPMENT  (in  Theme  42, Anomaly) 
r33.00,0.201PRQGRAM  BUDGET  (in  Theme  1 76, Popularity) 
r33.00,0.501ACQUISITIQN  EXECUTIVE  (in  Theme  176, Popularity) 
r32.00,1.001FAILURE  MODES  (in  Theme  68, Anomaly) 
r32.00,0.091SUPPQRT  REQUIREMENTS  (in  Theme  1 54, Emerging) 
r32.00,0.671ACQUISITIQN  DECISION  (in  Theme  149, Anomaly) 
r32.00.0.601CQST  ESTIMATES  (in  Theme  176, Popularity) 
r31.00,0.031DEVELQPMENT  PROCESS  (in  Theme  157, Popularity) 
r30.00.0.021SUPPQRT  SYSTEM  (in  Theme  169, Emerging) 
r30.00,1.001CHANGE  PROPOSALS  (in  Theme  21, Anomaly) 
r30.00.0.061TRAINING  SUPPORT  (in  Theme  179, Anomaly) 
r30.00,0.161PRQGRAM  OBJECTIVES  (in  Theme  1 76, Popularity) 
r30.00.0.031SYSTEMS  PERFORMANCE  (in  Theme  1 72, Popularity) 

[29.00, 0.24]TARGETCOST  (in  Theme  1 76, Popularity) 
r29.00,0.401LQGISTICS  AGENCY  (in  Theme  1 54, Emerging) 
r29.00,0. 1 31MAN AGEMENT  PROCESSES  (in  Theme  99, Emerging) 
r29.00,0.331ACQUISITIQN  SYSTEM  (in  Theme  157, Popularity) 
r29.00, 1 .001QUALITY  ASSURANCE  (in  Theme  46, Emerging) 
r29.00, 1 .001DEVELQPMENTAL  TESTING  (in  Theme  179, Anomaly) 
r29.00.0.331TEST  EVENTS  (in  Theme  179,Anomaly) 
r29.00,0.0510PERATIQNAL  REQUIREMENTS  (in  Theme  46, Emerging) 
r28.00. 1 .001MANPQWER  PERSONNEL  (in  Theme  106, Anomaly) 
r28.00,0.071SUPPQRT  STRATEGY  (in  Theme  166, Anomaly) 
r28.00.0.121CRITICAL  DESIGN  (in  Theme  68, Anomaly) 
r27.00,0.211CONTRQL  SYSTEM  (in  Theme  172, Popularity) 
r27.00,0.331REPAIR  PARTS  (in  Theme  141, Emerging) 
r27.00,0. 1 OIPRODUCT  LIFE  CYCLE  (in  Theme  169, Emerging) 

[27.00,0. 50]DEFENSEOSD  (in  Theme  166, Anomaly) 
[27.00,0.031INFORMATIQN  SYSTEMS  (in  Theme  46, Emerging) 

[27.00, 1.001KEY  INTERFACE  (in  Theme  99, Emerging) 

[27.00,0.1 1  PERFORMANCE  OBJECTIVES  (in  Theme  1 76, Popularity) 
r27.00.0.231TEST  PLANS  (in  Theme  1 72, Popularity) 
r27.00.0. 101JOINT  OPERATIONS  (in  Theme  1 69, Emerging) 

[26.00,0. 1 01SERVICE  SUPPORT  (in  Theme  15 7, Popularity) 
r26.00. 1 .001STRATEGIC  GUIDANCE  (in  Theme  75, Anomaly) 
[26.00,0.951MATERIAL  SHORTAGES  (in  Theme  99, Emerging) 
r26.00.0.051MAINTENANCE  SUPPORT  (in  Theme  1 69, Emerging) 
r26.00,0.041ENGINEERING  PROCESS  (in  Theme  15 7, Popularity) 
r26.00.0. 1 61SUSTAINMENT  REQUIREMENTS  (in  Theme  157, Popularity) 
[25.00,0.021SYSTEM  REQUIREMENTS  (in  Theme  1 54, Emerging) 
[25.00,0.071RISK  ASSESSMENT  (in  Theme  176, Popularity) 
[25.00,0.031MANAGEMENT  PROCESS  (in  Theme  15 7, Popularity) 
r25.00.0.181TEST  RESULTS  (in  Theme  52, Anomaly) 

[24.00,0.291DEFENSE  FEDERAL  (in  Theme  1 54, Emerging) 
[24.00,0.501SUPPQRT  COSTS  (in  Theme  120, Emerging) 
r24.00.0. 1 51PRODUCT  SUPPORT  STRATEGIES  (in  Theme  157,Popularity) 
[24.00,0.031MISSIQN  SYSTEM  (in  Theme  120, Emerging) 
r24.00.0.271HUMAN  FACTORS  (in  Theme  108, Anomaly) 

[24.00,0. 121CONTRACTOR  SUPPORT  (in  Theme  42, Anomaly) 
r23 .00,0.051 ACQUISITION  LIFE  CYCLE  (in  Theme  108, Anomaly) 

[23.00,0. 101SECTION  CHAPTER  (in  Theme  167, Popularity) 
[22.00,0.071TECHNICAL  SUPPORT  (in  Theme  46, Emerging) 

[22.00,0.1 1 IPROGRAM  EVALUATION  (in  Theme  1 04, Emerging) 
[22.00,0.071MANAGEMENT  INFORMATION  (in  Theme  46, Emerging) 

[2 1 .00,0.081PRQDUCT  SUPPORT  ELEMENTS  (in  Theme  1 1 1  Emerging) 

[2 1 .00,0.551TECHNICAL  MANUAL  (in  Theme  46, Emerging) 
[21.00,0.031TECHNICAL  REQUIREMENTS  (in  Theme  46, Emerging) 

[2 1  ■00,0.041PRQGRAM  COST  (in  Theme  1 76, Popularity) 

[2 1  ■00,0.0710PERAT1QNS  SUPPORT  (in  Theme  169, Emerging) 

[2 1 .00,0.201DESIGN  SERVICES  (in  Theme  1 79, Anomaly) 
[21.00,0.021GQVERNMENT  CONTRACT  (in  Theme  75, Anomaly) 

[2 1 .00,0.081SYSTEM  EQUIPMENT  (in  Theme  67, Anomaly) 

[2 1 .00, 1 .001ACQUISIT1QN  BOARD  (in  Theme  1 57, Popularity) 
r20.00,0.1 11SUPPORT  STRATEGIES  (in  Theme  15 7, Popularity) 
[20.00,0.321LQGISTICS  CENTER  (in  Theme  1 69, Emerging) 

[20.00, 1.001TITLE  SECTION  (in  Theme  1 67, Popularity) 
ri9.00,0.031REVIEW  PROCESS  (in  Theme  15 7, Popularity) 
f  1 9.00,0. 1 OICOMPONENT  ACQUISITION  (in  Theme  157, Popularity) 
ri9.00,0.491ANALYTICAL  TOOLS  (in  Theme  147, Anomaly) 

[1 9.00,0. 331LEVEL  ACTIVITIES  (in  Theme  154, Emerging) 
ri 9.00,0.0 11COST  PERFORMANCE  (in  Theme  1 76, Popularity) 
r  1 9.00.0.071PRQDUCT  DATA  (in  Theme  169, Emerging) 

[  1 9.00,0.671SERVICE  LIFE  (in  Theme  157, Popularity) 
ri8.00,0.751SYSTEMS  EQUIPMENT  (in  Theme  67, Anomaly) 
ri 8.00,0. 131FACTORS  ENGINEERING  (in  Theme  108, Anomaly) 
ri 8.00,0. 141PROGRAM  REVIEWS  (in  Theme  15 7, Popularity) 
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ri  8.00,0.2 11SUPP0RT  INFRASTRUCTURE  (in  Theme  99, Emerging) 
r  1 8.00.0.091PRQGRAM  DEVELOPMENT  (in  Theme  124, Anomaly) 
ri8.00,0.021SYSTEMS  MANAGEMENT  (in  Theme  1 72, Popularity) 
r  1 8.00.0.3 1  MAINTENANCE  PERSONNEL  (in  Theme  106, Anomaly) 
ri8.00,0.031ASSESSMENT  PROGRAM  (in  Theme  22, Anomaly) 
ri8.00,0.3710BSQLESCENCE  MANAGEMENT  (in  Theme  106, Anomaly) 
ri8.00,0.041TRAINlNG  SYSTEM  (in  Theme  179, Anomaly) 
ri7.00,0.331CHANGE  PROPOSAL  (in  Theme  21, Anomaly) 
r  1 7.00,0.251CQNTRACT  MODIFICATIONS  (in  Theme  1 54, Emerging) 
ri 7.00,0. 101PBL  CONTRACT  (in  Theme  149, Anomaly) 
r  1 7.00.0.031INTEGRATED  SYSTEM  (in  Theme  22, Anomaly) 
ri 7.00,0. 131CAPABILITIES  REQUIRED  (in  Theme  67, Anomaly) 
r  17.00.0.091 ACQUISITION  PHASE  (in  Theme  145, Anomaly) 
U6.00,0.071CQST  RISK  (in  Theme  1 76, Popularity) 

U 6.00,0.0 11SUPPORT  PROGRAM  (in  Theme  1 69, Emerging) 
ri 6.00,0. 141SUPPORT  ACTIVITY  (in  Theme  1 54, Emerging) 
ri 6.00,0. 101SUPPORT  ACTIVITIES  (in  Theme  1 54, Emerging) 
ri 6.00,0. 181SYSTEMS  ARCHITECTURE  (in  Theme  46, Emerging) 
U6.00,0.081PRQGRAM  INFORMATION  (in  Theme  46, Emerging) 
ri6.00,0.331PRQGRAM  REQUIREMENTS  (in  Theme  1 54, Emerging) 
r  1 6.00, 1  .OOIPERFORMANCE  PARAMETER  (in  Theme  46, Emerging) 
r  1 6.00, 1 .001MAINTENANCE  TASKS  (in  Theme  1 69, Emerging) 
U6.00,0.071PERFQRMANCE  REQUIREMENTS  (in  Theme  46, Emerging) 
ri5.00.0.711FIRM  FIXED  PRICE  (in  Theme  69,Anomaly) 
ri5.00,0.051MAJQR  SYSTEMS  (in  Theme  66, Anomaly) 
r  1 5.00, 1  .OOICONTROL  COMMUNICATIONS  (in  Theme  1 72,Popularity) 
U5.00,0.021DESIGN  SYSTEMS  (in  Theme  99, Emerging) 
IT5.00.0.051TRAINING  REQUIREMENTS  (in  Theme  179, Anomaly) 
ri5.00,0.231IMPRQVEMENT  PROGRAM  (in  Theme  120, Emerging) 

IT 5.00,0. 181TEST  EQUIPMENT  (in  Theme  1 76, Popularity) 
ri5.00,0.141DETAILED  DESIGN  (in  Theme  176, Popularity) 
U5.00.1.001MILITARY  HANDBOOK  (in  Theme  1 67,Popularity) 
U5.00,0.121PERFQRMANCE  METRICS  (in  Theme  15 7, Popularity) 
ri5.00,0.061SYSTEM  CONFIGURATION  (in  Theme  1 54, Emerging) 
[15.00,0.1 11SYSTEM  ARCHITECTURE  (in  Theme  46, Emerging) 
ri5.00.0.061TQTAL  COST  (in  Theme  68, Anomaly) 
r  1 5.00.0.071CAPABILITIES  DEVELOPMENT  (in  Theme  67, Anomaly) 
ri5.00,0.021LQGISTICS  REQUIREMENTS  (in  Theme  154, Emerging) 
ri5.00.0.081COMPQNENT  COST  (in  Theme  157, Popularity) 

U 5.00,0. 131QPERATIONAL  ISSUES  (in  Theme  46, Emerging) 
ri5.00.0.0611NTEGRATED  DATA  (in  Theme  22, Anomaly) 
ri5.00,0.031REQUIRED  SUPPORT  (in  Theme  67, Anomaly) 

IT 5.00,0. 1 UREPORTING  SYSTEM  (in  Theme  15 7, Popularity) 
U4.00,0.671QRDER  QUANTITY  (in  Theme  52, Anomaly) 
r  14.00,0. 161PROGRAM  COSTS  (in  Theme  120, Emerging) 
U4.00,0.371PRQDUCT  SUPPORT  ANALYTICAL  (in  Theme  147, Anomaly) 
r  14.00.0. 161AIR  SYSTEMS  (in  Theme  13, Anomaly) 
U4.00,0.091CONFIGURATIQN  CONTROL  (in  Theme  1 54, Emerging) 
U4.00,0.061CONTRACTQR  PERSONNEL  (in  Theme  106, Anomaly) 
r  14.00.0.331RESQURCES  REQUIRED  (in  Theme  155, Anomaly) 
U4.00,0.081DESIGN  OBJECTIVES  (in  Theme  1 76, Popularity) 
r  14.00.0.091MISSIQN  CRITICAL  (in  Theme  1 20, Emerging) 

U 4.00,0.2 11FIXED  PRICE  CONTRACTS  (in  Theme  69, Anomaly) 
r  14.00.0.201PERFQRMANCE  OUTCOMES  (in  Theme  149, Anomaly) 
ri4.00,0.411CQNTRACT  AWARDS  (in  Theme  167, Popularity) 
U4.00,0.051SUPPLY  MANAGEMENT  (in  Theme  168, Anomaly) 
U4.00,0.281CQNTRACT  PRICING  (in  Theme  149, Anomaly) 
U4.00,0.141MAINTENANCE  DATA  (in  Theme  1 69, Emerging) 
ri3.00,0.041RELlABILITY  MANAGEMENT  (in  Theme  153, Anomaly) 
fl 3.00,0 .241INDUSTRIAL  FACILITIES  (in  Theme  1 69, Emerging) 
r  1 3 .00,0.0 1 1ENG1NEERING  MANAGEMENT  (in  Theme  108, Anomaly) 
ri3.00,0.031DEFENSE  CONTRACT  (in  Theme  124, Anomaly) 
ri3.00,0.041SUPPQRT  PLANNING  (in  Theme  120, Emerging) 
ri3.00,0.501SUPPQRT  SERVICES  (in  Theme  179, Anomaly) 
f  1 3.00,0.331CQNTRACT  ACTION  (in  Theme  104, Emerging) 
fl 3.00,0. 131GENERAL  PURPOSE  (in  Theme  42, Anomaly) 
U3.00,0.061FEDERAL  PROCUREMENT  (in  Theme  104, Emerging) 
ri3.00,0.051SUPPQRT  COST  (in  Theme  1 69, Emerging) 
ri3.00,0.501ACQUISITIQN  STRATEGIES  (in  Theme  15 7, Popularity) 
ri3.00,0.051TQTAL  SYSTEM  (in  Theme  68, Anomaly) 
r  1 3 .00,0. 1 71  WORKING  LEVEL  (in  Theme  1 54, Emerging) 
ri3.00,0.611IA  CONTROLS  (in  Theme  32, Anomaly) 
ri3.00,0.221HIGHER  LEVEL  (in  Theme  147, Anomaly) 
fl 3.00,0. ^REPRESENTATIVE  TEST  (in  Theme  38, Anomaly) 
ri 3.00.0. 121SOFTWARE  DOCUMENTATION  (in  Theme  179, Anomaly) 
fl 3.00,0. 121SYSTEM  FUNCTIONAL  (in  Theme  120, Emerging) 
ri3.00,0.031PRQGRAM  LIFE  CYCLE  (in  Theme  108, Anomaly) 
U3.00,0.031DESIGN  ENGINEERING  (in  Theme  99, Emerging) 
fl 3.00,0.01 1DATA  REQUIREMENTS  (in  Theme  46, Emerging) 
IT2.00.0.071SUPPLY  SYSTEM  (in  Theme  168, Anomaly) 
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ri2.00,0.031SUPPQRT  CONTRACTS  (in  Theme  131, Anomaly) 
r  12.00,0.071RESQURCES  SUPPORT  (in  Theme  155, Anomaly) 
ri2.00,0.091SYSTEMS  APPROACH  (in  Theme  67,Anomaly) 
ri2.00.0.091DATA  REPORTING  (in  Theme  15 7, Popularity) 
ri2.00,0.071CQST  ELEMENTS  (in  Theme  1 1 1, Emerging) 
ri2.00,0.401PRQGRAM  GOALS  (in  Theme  1 76,Popularity) 
r  12.00, 1.00HNDEPENDENT  COST  (in  Theme  1 76, Popularity) 
ri 2.00,0. 101EARLY  OPERATIONAL  (in  Theme  149, Anomaly) 
ri2.00,0.011MAINTENANCE  MANAGEMENT  (in  Theme  1 69, Emerging) 
ri2.00,0.331PRQGRAM  RISK  (in  Theme  1 76, Popularity) 
r  12.00.0.451DEVELQPMENT  TEST  (in  Theme  1 76, Popularity) 
r  1 2.00,0. 121EFFECTIVE  SUPPORT  (in  Theme  99, Emerging) 
ri 2.00.0.0 11LOGIST1CS  INFORMATION  (in  Theme  1 69, Emerging) 
ri2.00,0.051PHASE  SUSTAINMENT  (in  Theme  145, Anomaly) 
ri 2.00,0. 171CONTRACT  TERMS  (in  Theme  21, Anomaly) 
ri2.00,0.071SYSTEMS  CENTER  (in  Theme  1 69, Emerging) 

IT2.00.0.381TYPE  CONTRACTS  (in  Theme  1 69, Emerging) 
ri2.00,0.381MISSIQN  FAILURES  (in  Theme  68, Anomaly) 
ri2.00,0.071SQFTWARE  DEVELOPMENT  (in  Theme  1 69, Emerging) 
ri2.00,0. 1 51DMSMS  MANAGEMENT  (in  Theme  99, Emerging) 
ri2.00,0.021TEST  PROGRAM  (in  Theme  1 76, Popularity) 

[11 .00,0.0 11LOGISTICS  MANAGEMENT  (in  Theme  1 69, Emerging) 

[11  ■00,0.021MANAGEMENT  PLANNING  (in  Theme  120, Emerging) 
ri  1.00. 1.001 ACQUISITION  RESOURCES  (in  Theme  155, Anomaly) 

[1 1.00,0.561QUANTITY  EOQ  (in  Theme  52, Anomaly) 

U  1.00.0.401MANAGEMENT  AGENCY  (in  Theme  1 54, Emerging) 

[11 .00,0.071RELIABILITY  ENGINEERING  (in  Theme  108, Anomaly) 

U  1.00.0.1 51  ANALYSIS  IMPROVEMENT  (in  Theme  120, Emerging) 

[1 1.00,0.051RELATED  DESIGN  (in  Theme  1 69, Emerging) 

[  1 1 .00,0. 1 31TECHN1CAL  DOCUMENTATION  (in  Theme  1 79, Anomaly) 

[11 .00, 1 .001TECHNICAL  ARCHITECTURE  (in  Theme  46, Emerging) 

U1 .00.0.1 21QRGANIC  SUPPORT  (in  Theme  169, Emerging) 

[1 1.00,0.201ENVIRQNMENTAL  SAFETY  (in  Theme  66, Anomaly) 
[11.00,0.461FIXED  PRICE  INCENTIVE  (in  Theme  1 69, Emerging) 

[11 .00,0.031REQUIRED  DATA  (in  Theme  67, Anomaly) 

[11 .00.0.131FIXED  PRICE  CONTRACT  (in  Theme  69, Anomaly) 

[11 .00.0.021ANALYSIS  PROCESS  (in  Theme  157, Popularity) 

[1  l.OO.O.OULOGISTICS  PROGRAM  (in  Theme  169, Emerging) 

U  1.00,0.1 1  PROGRAM  PLANS  (in  Theme  172, Popularity) 

[11 .00,0.501DETAILED  TEST  (in  Theme  1 76, Popularity) 

[11 .00,0.131FULL  OPERATIONAL  (in  Theme  149, Anomaly) 

[1 1.00,0.331FACILITIES  MAINTENANCE  (in  Theme  169, Emerging) 

U  1.00.0.021SYSTEM  TESTING  (in  Theme  179, Anomaly) 

[  1 1 .00,0. 1 71DATA  ANALYSIS  (in  Theme  1 5, Anomaly) 

[11  ■00,0.071MAJQR  PROGRAM  (in  Theme  66, Anomaly) 
[10.00,0.06PRQGRAM  LEVEL  (in  Theme  106, Anomaly) 
riQ.00.0. 1 11FAILURE  REPORTING  (in  Theme  15 7, Popularity) 
riO.OO.l.OOlSERVICE  ACQUISITION  (in  Theme  157, Popularity) 
ri0.00,0.021SUSTAINMENT  SYSTEMS  (in  Theme  15 7, Popularity) 
ri0.00,0.071CQST  OBJECTIVES  (in  Theme  1 76, Popularity) 
ri 0.00,0. 171BUSINESS  STRATEGY  (in  Theme  124, Anomaly) 
r  10.00.0.1 11DESIGN  CONSIDERATIONS  (in  Theme  1 54, Emerging) 
riO.OO.O.lOlTEST  OBJECTIVES  (in  Theme  1 76, Popularity) 
[10.00,0.03]REQUIRED  PERFORMANCE  (in  Theme  67, Anomaly) 

[10.00  0.301SYSTEMS  ENGINEER  (in  Theme  153, Anomaly) 

[1 0.00,0. 191HUMAN  PERFORMANCE  (in  Theme  108, Anomaly) 
ri0.00,0.041SPECIFIC  SYSTEM  (in  Theme  120, Emerging) 
ri0.00,0.271DATA  MANAGERS  (in  Theme  42, Anomaly) 

[1 0.00,0. 181FUNCT1QNAL  REQUIREMENTS  (in  Theme  120, Emerging) 
[10.00,0.081LIFE  CYCLE  ENGINEERING  (in  Theme  108, Anomaly) 
r  1 0.00, 1  ^PERFORMANCE  CHARACTERISTICS  (in  Theme  46, Emerging) 
ri0.00,0.30PRQGRAM  FUNDING  (in  Theme  1 76,Popularity) 

[  1 0.00, 1  .OOICOST  CONTROL  (in  Theme  1 72, Popularity) 
ri 0.00,0. 19PINAL  COST  (in  Theme  15, Anomaly) 
ri0.00,0.061SPECIAL  EQUIPMENT  (in  Theme  1 69, Emerging) 
ri0.00,0.021GQVERNMENT  CONTRACTOR  (in  Theme  42, Anomaly) 
[10.00,0.261CONTINUQUS  PROCESS  (in  Theme  141, Emerging) 
ri0.00,0.081TEST  SUPPORT  (in  Theme  1 69, Emerging) 

[9.00,0.501TRAINING  MATERIALS  (in  Theme  179, Anomaly) 

[9.00,0. 151PERFORMANCE  MEASURES  (in  Theme  32, Anomaly) 

[9.00,0. 131MATERIAL  AVAILABILITY  (in  Theme  15 7, Popularity) 
[9.00,0.05PRQV1DE  ADDITIONAL  (in  Theme  1 76, Popularity) 
[9.00,0.051EQU1PMENT  REPAIR  (in  Theme  141, Emerging) 
[9.00,0.671BUSINESS  CONCERNS  (in  Theme  124, Anomaly) 
r9.00.0.081DATA  ENVIRONMENT  (in  Theme  22, Anomaly) 
r9.00.0.061CQST  ANALYSIS  (in  Theme  1 76, Popularity) 
[9.00,0.021SERVICES  REQUIRED  (in  Theme  179, Anomaly) 
[9.00,0.071FUNCTIQNS  REQUIRED  (in  Theme  1 1 1, Emerging) 
[9.00,0.391NAVAL  AIR  (in  Theme  1 69, Emerging) 
r9.00.0.071CONSTRUCTIQN  CONTRACTS  (in  Theme  1 1 1, Emerging) 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-348- 


[9.00,1.00]SUPPQRT  CENTER  (in  Theme  169, Emerging) 
r9.00.0.17HI  PROGRAMS  (in  Theme  145,Anomaly) 

[9.00,0. 101MANAGEMENT  STRUCTURE  (in  Theme  1 1 1, Emerging) 
r9.00.0.031MATERIAL  EQUIPMENT  (in  Theme  99, Emerging) 
[9.00,0.021DATA  STRATEGY  (in  Theme  166, Anomaly) 

[9.00,0. 121ELECTRONIC  TECHNICAL  (in  Theme  172, Popularity) 
[9.00,0.0511NFQRMAT10N  OPERATIONS  (in  Theme  169, Emerging) 
r9.00,0.041MATERIAL  SUPPORT  (in  Theme  99, Emerging) 
[9.00,0.041SUSTAINMENT  PROGRAM  (in  Theme  157, Popularity) 
[9.00,1.001PERFQRMANCE  RELIABILITY  (in  Theme  153, Anomaly) 
r9.00.0.06HNTERFACE  REQUIREMENTS  (in  Theme  99, Emerging) 
r9.00,0.071TRAINING  FACILITIES  (in  Theme  1 69, Emerging) 
r9.00.1.001EXCHANGE  DATA  (in  Theme  46, Emerging) 
r9.00,0.071SUSTAINMENT  STRATEGIES  (in  Theme  15 7, Popularity) 
[9.00.0.231DATA  BASE  (in  Theme  46, Emerging) 

[9.00,0.081STANDARDS  PERFORMANCE  (in  Theme  32, Anomaly) 
[9.00.0.071MANAGER  LOGISTICS  (in  Theme  108, Anomaly) 
[9.00,0.081CURRENT  ACQUISITION  (in  Theme  157, Popularity) 
[9.00,0.061SYSTEM  SOFTWARE  (in  Theme  1 69, Emerging) 
r9.00,0.251DEVELQPMENT  MODEL  (in  Theme  179, Anomaly) 

[9.00,0.1 11RESOURCE  MANAGEMENT  (in  Theme  95, Anomaly) 
[9.00,0.02]SYSTEMDATA  (in  Theme  46, Emerging) 

[9.00,0.051MISSIQN  PERFORMANCE  (in  Theme  120, Emerging) 
r9.00.0.031LQGISTICS  PRODUCT  (in  Theme  1 69, Emerging) 

[9.00,0. 17]DOD_WIDE  (in  Theme  32, Anomaly) 

[9.00,0.0  l]SUPPORTDAIA  (in  Theme  46, Emerging) 
[8.00,0.331DEVELQPMENT  EFFORTS  (in  Theme  1 54, Emerging) 
[8.00.0.011ACOUISITIQN  CONTRACTS  (in  Theme  131, Anomaly) 
[8.00,0.091ACTIQN  SYSTEM  (in  Theme  1 04, Emerging) 
[8.00,0.051REPQRTING  REQUIREMENTS  (in  Theme  15 7, Popularity) 
[8.00,0.0410PERATIQNAL  PERFORMANCE  (in  Theme  46, Emerging) 
[8.00.0.28TTECHNICAL  AUTHORITY  (in  Theme  46, Emerging) 

[8.00,0. 131DOCUMENTATION  SUPPORT  (in  Theme  179, Anomaly) 
[8.00,0. 141CHANGE  MANAGEMENT  (in  Theme  2 1 , Anomaly) 
[8.00,0.071EXISTING  PROGRAM  (in  Theme  21, Anomaly) 
[8.00.0.011DESIGN  REQUIREMENTS  (in  Theme  99, Emerging) 
[8.00.0.041MANAGEMENT  APPROACH  (in  Theme  67, Anomaly) 
[8.00,1.001INFORMATIQN  EXCHANGE  (in  Theme  46, Emerging) 

[8.00,0. 171CONTRACT  DELIVERY  (in  Theme  155, Anomaly) 
[8.00,0.0nSYSTEMS  PROGRAM  (in  Theme  172, Popularity) 

[8.00.1  .OOIDEVELOPMENT  ACTIVITIES  (in  Theme  154, Emerging) 
[8.00,0.051PRQVIDE  EFFECTIVE  (in  Theme  99, Emerging) 
[8.00.0.201SUPPQRT  AGREEMENTS  (in  Theme  153, Anomaly) 

[8 .00,0. 161  VERIFICATION  TESTING  (in  Theme  70, Anomaly) 
[8.00,0.041LQGISTICS  RELATED  (in  Theme  1 69, Emerging) 
[8.00,0.041PRQGRAM  CHANGE  (in  Theme  21, Anomaly) 
[8.00.0.051SUPPQRTING  DATA  (in  Theme  95, Anomaly) 
[8.00,0.351ENGINEERING  SE  (in  Theme  166, Anomaly) 
[8.00,0.331SPECIFICATIQN  REQUIREMENTS  (in  Theme  1 72, Popularity) 
[8.00.0.1 11SPECIAL  CONTRACT  (in  Theme  1 69, Emerging) 
[8.00,0.241DEVELQPMENT  EMD  (in  Theme  124, Anomaly) 
[8.00.0.031PRQGRAM  MILESTONE  (in  Theme  145, Anomaly) 

[8.00,0. 111SYSTEM  RELIABILITY  (in  Theme  153, Anomaly) 
[8.00.0.061EXISTING  CONTRACTS  (in  Theme  21, Anomaly) 
[8.00,0.021MANAGEMENT  CONTROL  (in  Theme  1 72, Popularity) 
[8.00,0.071MAINTENANCE  PLANS  (in  Theme  1 72, Popularity) 
[8.00,0.021RELIABIL1TY  PROGRAM  (in  Theme  153, Anomaly) 
[8.00,0.011PERSQNNEL  REQUIREMENTS  (in  Theme  106,Anomaly) 
[8.00,0.011SQFTWARE  SYSTEMS  (in  Theme  169, Emerging) 
[8.00,0.081GQVERNMENT  ORGANIZATION  (in  Theme  1 5 4, Emerging) 
[8.00,0.331SUPPQRT  CONCEPTS  (in  Theme  179, Anomaly) 
[8.00,1.001RELATED  SUPPORT  (in  Theme  1 69, Emerging) 
[8.00,0.021ENGINEERING  PROGRAM  (in  Theme  108, Anomaly) 

[8.00,0. 101ACOUISITION  COST  (in  Theme  15 7, Popularity) 
[8.00,0.041LQGISTICS  PMS  (in  Theme  104, Emerging) 

[8.00,0.091SYSTEM  HARDWARE  (in  Theme  67, Anomaly) 

[8.00,0. 141RELATED  ISSUES  (in  Theme  1 69, Emerging) 

[8.00,1.001CQST  CATEGORIES  (in  Theme  172, Popularity) 
[7.00,0.041EQUIPMENT  REQUIRED  (in  Theme  67, Anomaly) 
[7.00,0.021MAINTENANCE  PROGRAM  (in  Theme  169, Emerging) 

[7.00,0. 181MANUFACTURING  TECHNOLOGY  (in  Theme  1 57, Popularity) 
[7. 00,0.091 VERIFICATION  REQUIREMENTS  (in  Theme  70, Anomaly) 
[7.00,0.201WARFIGHTER  REQUIREMENTS  (in  Theme  149, Anomaly) 
[7.00,0.051CURRENT  PROGRAM  (in  Theme  15 7, Popularity) 
[7.00.0.041PERFQRMANCE  DATA  (in  Theme  46, Emerging) 

[7.00.0. 101APPROVED  ACQUISITION  (in  Theme  75, Anomaly) 
[7.00,0.041PERSQNNEL  TRAINING  (in  Theme  106, Anomaly) 
[7.00.0.221CRITICAL  SYSTEM  (in  Theme  68, Anomaly) 
[7.00,0.021ENGINEERING  DESIGN  (in  Theme  108, Anomaly) 
[7.00.0.031MANAGEMENT  STRATEGY  (in  Theme  1 15, Anomaly) 
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[7.00,0.031BASED  RELIABILITY  (in  Theme  157, Popularity) 
r7.00.0.201AIRCRAFT  SYSTEMS  (in  Theme  172, Popularity) 
r7.00,0.121MATERIEL  SYSTEM  (in  Theme  131, Anomaly) 
r7.00.0.501ACOUlSlTIQN  APPROACH  (in  Theme  67, Anomaly) 
r7.00,0.001ACQUISITIQN  CONTRACT  (in  Theme  167, Popularity) 
r7.00,0.021PLANNING  PROGRAM  (in  Theme  1 20, Emerging) 
r7.00,0.061RELATED  DATA  (in  Theme  169, Emerging) 
r7.00,0.291DEVELQPMENT  MODELS  (in  Theme  153, Anomaly) 
r7.00, 1 .001MISSIQN  ACCOMPLISHMENT  (in  Theme  120, Emerging) 
r7.00,0.031MAINTENANCE  PROCESSES  (in  Theme  99, Emerging) 
r7.00.0.041SUPPQRT ABILITY  REQUIREMENTS  (in  Theme  108, Anomaly) 
r7.00,0.021SUPPQRT  COST  (in  Theme  176, Popularity) 
r7.00.0.081DMSMS  PROGRAM  (in  Theme  99, Emerging) 
r7.00, 1 .001MANUFACTURING  SOURCES  (in  Theme  1 57, Popularity) 
r7.00.0.071MILlTARY  STANDARD  (in  Theme  1 67, Popularity) 
r7.00, 1  .OOIPROVIDE  DETAILED  (in  Theme  176, Popularity) 
r7.00.0.331REPAlR  WORKLOADS  (in  Theme  141, Emerging) 
r7.00,0.011PERSQNNEL  EQUIPMENT  (in  Theme  106, Anomaly) 
r7.00,0.031CONTRQL  SYSTEMS  (in  Theme  172, Popularity) 
r7.00,0.201DQD  POLICIES  (in  Theme  1 76, Popularity) 
r7.00,0.331ENGINEERING  LOGISTICS  (in  Theme  1 69, Emerging) 

17.00,0. 121ELECTRONIC  SYSTEMS  (in  Theme  172, Popularity) 
r7.00,0.051FEE  CONTRACT  (in  Theme  46, Emerging) 
f7. 00.0.0 llSOFTWARE  MANAGEMENT  (in  Theme  169,Emerging) 
r7.00,0.041MISSIQN  REQUIREMENTS  (in  Theme  120, Emerging) 
r6.00.0.051MAINTENANCE  COSTS  (in  Theme  120, Emerging) 
r6.00,0.011GQVERNMENT  PROGRAM  (in  Theme  75, Anomaly) 
r6.00,0.331FMECA  FAILURE  (in  Theme  68, Anomaly) 
r6.00,0.291REQUIREMENTS  DETERMINATION  (in  Theme  155, Anomaly) 
r6.00,0.081FUTURE  CONTRACTS  (in  Theme  176, Popularity) 
r6.00,1.001CHAIRMAN  JOINT  (in  Theme  95, Anomaly) 
r6.00,0.021PBL  SUPPORT  (in  Theme  149, Anomaly) 
r6.00,0.011RELlABlLlTY  ANALYSIS  (in  Theme  153, Anomaly) 
r6.00,0.031TECHNICAL  RISK  (in  Theme  1 76, Popularity) 
[6.00.0.071SOFTWARE  RELATED  (in  Theme  1 69, Emerging) 
r6.00.0. 121INTEROPERABILITY  TESTING  (in  Theme  70, Anomaly) 
r6.00.0.101PERFQRMANCE  SPECIFICATIONS  (in  Theme  108, Anomaly) 
r6.00,0.061DOCUMENTATIQN  REQUIRED  (in  Theme  179, Anomaly) 
r6.00,0.061REQUlREMENTS  DEFINITION  (in  Theme  1 67, Popularity) 
r6.00,0.011LQGISTICS  PLANNING  (in  Theme  120, Emerging) 
r6.00.1.001MISSIQN  ESSENTIAL  (in  Theme  120, Emerging) 
r6.00,0.101GQVERNMENT  PURPOSE  (in  Theme  42, Anomaly) 
r6.00.0.021SERVlCES  CONTRACT  (in  Theme  179, Anomaly) 

[6.00, 1 .00]INCENTIVETYPE  (in  Theme  1 69, Emerging) 
r6.00,0.031REQUIREMENTS  DOCUMENTS  (in  Theme  56,Anomaly) 
r6.00,0.031FAClLITlES  ENGINEERING  (in  Theme  1 69, Emerging) 
r6.00.0.0nSUPPQRT  PERFORMANCE  (in  Theme  169, Emerging) 
r6.00,0.09:]TERM  TECHNICAL  (in  Theme  108, Anomaly) 
r6.00,0.081VEHICLE  SYSTEMS  (in  Theme  172,Popularity) 

[6.00.0.451TEST  PROCEDURE  (in  Theme  75, Anomaly) 
r6.00,0.251PRQGRAM  APPROVED  (in  Theme  75, Anomaly) 
r6.00.0.331ACOUISlTIQN  RELATED  (in  Theme  1 69, Emerging) 
r6.00,0.601EMD  PHASE  (in  Theme  124, Anomaly) 
r6.00.0.061SYSTEM  STRUCTURE  (in  Theme  1 1 1, Emerging) 
r6.00,0.051MANAGEMENT  PLANS  (in  Theme  172, Popularity) 
r6.00,0.071REQUIRES  PROGRAM  (in  Theme  167, Popularity) 
[6.00,0.041RELATED  ACTIVITIES  (in  Theme  1 69, Emerging) 
r6.00,0.141QVERSIGHT  PROGRAM  (in  Theme  95, Anomaly) 
r6.00,0.071SUPPQRT  CONTRACTORS  (in  Theme  108, Anomaly) 
r6.00,0.061INDIVIDUAL  TRAINING  (in  Theme  106, Anomaly) 
r6.00,0.201DEVELQPMENT  APPROACH  (in  Theme  67, Anomaly) 
r6.00,0.011PRQGRAM  PERFORMANCE  (in  Theme  46, Emerging) 
r6.00,0. 141FAILURE  CRITERIA  (in  Theme  68,Anomaly) 
r6.00,0.021PRQGRAM  DESIGN  (in  Theme  99, Emerging) 
r6.00,0.081MANAGEMENT  WORKING  (in  Theme  1 54, Emerging) 
r6.00,0.021EQUIPMENT  RELIABILITY  (in  Theme  153,Anomaly) 
r6.00,0.031SQFTWARE  DATA  (in  Theme  1 69, Emerging) 
r6.00,0.501PRQPRIETARY  INFORMATION  (in  Theme  46, Emerging) 
r6.00,0.101DESIGN  EFFORTS  (in  Theme  1 54, Emerging) 
r6.00,0.501WAlVER  REQUEST  (in  Theme  148, Anomaly) 
r6.00,0.081EFFECTIVE  MAINTENANCE  (in  Theme  99, Emerging) 
r6.00,0.061COMPQNENT  TESTING  (in  Theme  157,Popularity) 
r6.00,0.021RELATED  INFORMATION  (in  Theme  169, Emerging) 
f5 .00,0.1 11CRITICAL  FAILURES  (in  Theme  68, Anomaly) 
r5.00.0.401PEQ  PROGRAM  (in  Theme  1 04, Emerging) 
r5.00.0.031BASED  APPROACH  (in  Theme  67, Anomaly) 
r 5 .00,0.061EQUIPMENT  RELATED  (in  Theme  1 69, Emerging) 
r5.00,0.071BASELINE  DESIGN  (in  Theme  1 76, Popularity) 
r5.00,1.001CONSTRUCTIQN  CONTRACT  (in  Theme  1 67, Popularity) 
r5.00.0.031USC  SECTION  (in  Theme  1 67, Popularity) 
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r 5 .00,0.071SUPPQRT  FACILITIES  (in  Theme  1 69, Emerging) 
r5.00.0.031SPECIFIC  ELEMENTS  (in  Theme  1 1 1, Emerging) 

\5 .00,0.0 11TIME  BASED  (in  Theme  141, Emerging) 
r5.00.0.061NAVY  TEST  (in  Theme  124, Anomaly) 
r 5 .00,0.081MAINTAINABILITY  PROGRAM  (in  Theme  153, Anomaly) 
f5 .00,0.041 VEHICLE  CONTROL  (in  Theme  172, Popularity) 
[5.00,0.061SUPPQRT  ARCHITECTURE  (in  Theme  46, Emerging) 
r5.00,1.001ANALYSIS  TOOL  (in  Theme  15, Anomaly) 

[5 .00,0. 1 1 1DATA  MODEL  (in  Theme  1 79, Anomaly) 
r5.00,0.071DEFENSE  RESEARCH  (in  Theme  124, Anomaly) 
r5 .00.0.01  IS YSTEM  EVALUATION  (in  Theme  104, Emerging) 
r 5 .00,0.051STRATEGY  MAINTENANCE  (in  Theme  166, Anomaly) 
r5.00.0.051HUMAN  ENGINEERING  (in  Theme  108, Anomaly) 
r 5 .00,0.071MANAGEMENT  ORGANIZATION  (in  Theme  1 54, Emerging) 
r5. 00.0.3 11FUNCTIONAL  PERFORMANCE  (in  Theme  120, Emerging) 
r5.00,0.171DQD  JOINT  (in  Theme  95, Anomaly) 
r5.00.0.021CQSTS  PERFORMANCE  (in  Theme  120, Emerging) 
r5.00,0.031GQVERNMENT  ACTIVITIES  (in  Theme  1 54, Emerging) 
r5.00,0.051SUSTAINMENT  APPROACH  (in  Theme  67, Anomaly) 
r5.00,0.131REDUCE  PROGRAM  (in  Theme  1 76, Popularity) 
f5 .00,0. 1 71INITIAL  CONTRACT  (in  Theme  106, Anomaly) 
[5.00.0.071MANAGEMENT  TOOLS  (in  Theme  147, Anomaly) 
r 5 .00,0.03 1TECHNICAL  APPROACH  (in  Theme  67, Anomaly) 
r 5  .OO.O.Q61PROPOSED  ACQUISITION  (in  Theme  1 76, Popularity) 
r 5 .00.0.501GATHERING  DATA  (in  Theme  46, Emerging) 
r5.00.1.001TEST  PROCEDURES  (in  Theme  147, Anomaly) 
r5.00,0.331TECHNICAL  PARAMETER  (in  Theme  46, Emerging) 
r5 .00.0.021MAINTENANCE  CAPABILITY  (in  Theme  32, Anomaly) 
r 5 .00.0.091ACTION  ITEMS  (in  Theme  104, Emerging) 
r5.00,0.041SUSTAINMENT  EFFORTS  (in  Theme  1 54, Emerging) 
r5.00,0.081S YSTEM  SPECIFICATION  (in  Theme  1 72, Popularity) 
r5.00.0.331SYSTEM  CHARACTERISTICS  (in  Theme  46,Emerging) 
r5.00,0.091AWARD  CONTRACTS  (in  Theme  167, Popularity) 

\5 .00,0.01  IPROGRAM  STRATEGY  (in  Theme  166, Anomaly) 
r5.00,0.031MAINTAIN ABILITY  DESIGN  (in  Theme  153, Anomaly) 
r5 .00.0.251ENGINEERING  DEVELOPMENT  (in  Theme  108, Anomaly) 
r5 .00.0.0 11PERFORMANCE  MANAGEMENT  (in  Theme  46, Emerging) 
f5 .00,0. 1 31PERSONNEL  NEEDED  (in  Theme  141, Emerging) 
r5.00.0.061SYSTEMS  PROVIDING  (in  Theme  99, Emerging) 
r 5 .00,0.061DESIGN  REVIEWS  (in  Theme  157, Popularity) 
r5.00.1.001TECHNOLQGY  STANDARDS  (in  Theme  32, Anomaly) 
f5 .00, 1 .001PRODUCTIQN  FACILITIES  (in  Theme  1 69, Emerging) 
r 5 .00.0.061PRIMARY  SUPPORT  (in  Theme  15 7, Popularity) 
r 5 .00,0.051DEVELQPMENT  EFFORT  (in  Theme  141, Emerging) 
r 5 .00,0.041INCENTI VE  CONTRACT  (in  Theme  1 69, Emerging) 
r5.00,0.071SYSTEM  ELEMENTS  (in  Theme  1 1 1, Emerging) 
r5.00.0.021ANALYSlS  REQUIRED  (in  Theme  67, Anomaly) 
r5.00,0.671INlTIAL  OPERATIONAL  (in  Theme  46, Emerging) 
r5 .00,0.3 11REPAIRABLE  ITEMS  (in  Theme  91, Anomaly) 
r5.00.0.041PACKAGE  PROCUREMENT  (in  Theme  104, Emerging) 
r 5 .00,0.051ALTERNATIVE  DESIGN  (in  Theme  15 7, Popularity) 
r5 .00,0. 1 71PERSONNEL  SURVIVABILITY  (in  Theme  106, Anomaly) 
r 5 .00,0.501REPAIRABLE  ITEM  (in  Theme  91, Anomaly) 
r5.00.0.121SUCCESSFUL  MISSION  (in  Theme  120, Emerging) 
r 5 .00,0.041SQFT WARE  CONFIGURATION  (in  Theme  1 69, Emerging) 
r 5 .00,0.071ADMINISTRATI VE  CONTROL  (in  Theme  66, Anomaly) 
r 5 .00,0.071PLANNING  DOCUMENTS  (in  Theme  56, Anomaly) 
r5.00,0.141SKILLS  REQUIRED  (in  Theme  67,Anomaly) 
r5.00,0.071SPECIFlCAT10NS  ENGINEERING  (in  Theme  108, Anomaly) 
\5 .00,0. 131SOFT WARE  RESOURCES  (in  Theme  155, Anomaly) 

\5 .00,0.0 11ENGINEERING  TECHNICAL  (in  Theme  46, Emerging) 
r5.00,0.081IA  PROGRAM  (in  Theme  108, Anomaly) 
r5.00,0.061CQSTS  RELATED  (in  Theme  169, Emerging) 
r 5 .00,0.021CRITICAL  MAINTENANCE  (in  Theme  68, Anomaly) 

\5 .00,0.0 11PERSONNEL  MANAGEMENT  (in  Theme  106, Anomaly) 

\5 .00,0.01 1SUSTAINMENT  LOGISTICS  (in  Theme  157, Popularity) 
r 5 .00,0.051REPQRTING  PROCESS  (in  Theme  157, Popularity) 
r5.00,0.0310PERATIQNS  CENTER  (in  Theme  1 69, Emerging) 
r 5 .00,0.021EVALUATIQN  STRATEGY  (in  Theme  1 04, Emerging) 
r5.00,0.001SUPPQRT  MANAGEMENT  (in  Theme  1 69, Emerging) 
r 5 .00,0.041QPERATING  SUPPORT  (in  Theme  1 76, Popularity) 
r4.00,0.021SYSTEM  CAPABILITIES  (in  Theme  67, Anomaly) 
r4.00,0.021LEVEL  REQUIREMENTS  (in  Theme  106, Anomaly) 
r4.00,0. 1 31SUPPORT  SOLUTION  (in  Theme  149, Anomaly) 
r4.00.0.031CQNTRACT  DATA  (in  Theme  167, Popularity) 
r4.00.0.011LQGISTICS  STRATEGY  (in  Theme  166, Anomaly) 
r4.00,0.011TEST  SYSTEM  (in  Theme  1 76, Popularity) 
r4.00,0.031MISSIQN  CAPABILITY  (in  Theme  120, Emerging) 
r4.00,0.201DESIGN  CONCEPTS  (in  Theme  179, Anomaly) 

[4.00,0. 141ITEMS  REQUIRING  (in  Theme  106, Anomaly) 
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r4.00,0.1 11MAINTENANCE  STRATEGIES  (in  Theme  15 7, Popularity) 
KOO.O.OnSYSTEM  MAINTENANCE  (in  Theme  1 69, Emerging) 
r4.00,0.071KEY  SUSTAINMENT  (in  Theme  99, Emerging) 
r4.00.0.091PBL  APPROACH  (in  Theme  149, Anomaly) 
r4.00,0.051FUNCTIQNAL  RESPONSIBILITIES  (in  Theme  66, Anomaly) 
[4.00,0.01  ICONTRACT  DEVELOPMENT  (in  Theme  1 67, Popularity) 
r4.00, 1 .001DISCREPANC Y  REPORTS  (in  Theme  168, Anomaly) 
r4.00,0.041LIMITED  BASED  (in  Theme  32,Anomaly) 
r4.00,0.001LQGlSTICS  SYSTEM  (in  Theme  169, Emerging) 

[4.00,0.091DATA  NEEDED  (in  Theme  141, Emerging) 
r4.00.0.021SUPPQRT  ITEMS  (in  Theme  91, Anomaly) 
r4.00,0.041SPECIFIC  MAINTENANCE  (in  Theme  120, Emerging) 
r4.00.0.021DEVELQPMENT  TESTING  (in  Theme  179, Anomaly) 
r4.00,0.021LEVEL  RISK  (in  Theme  1 76, Popularity) 
r4.00.0.201MEASUREMENT  SYSTEM  (in  Theme  172, Popularity) 
r4.00,0.031PRQGRAM  STRUCTURE  (in  Theme  1 1 1, Emerging) 
r4.00.0.201SYSTEM  APPROACH  (in  Theme  67, Anomaly) 
[4.00,0.021SUPPQRT  CONFIGURATION  (in  Theme  1 54, Emerging) 
r4.00,0.031DEPLQYMENT  SUPPORT  (in  Theme  1 54, Emerging) 

[4.00,0.0 1 1EVALUATION  MANAGEMENT  (in  Theme  1 04, Emerging) 
r4.00,0.331AFFQRDABLE  COST  (in  Theme  1 15, Anomaly) 
r4.00,0.251SYSTEM  FUNCTIONALITY  (in  Theme  172, Popularity) 
r4.00,0.081PRQGRAM  DECISIONS  (in  Theme  149, Anomaly) 
r4.00.0.051NAVY  MODELING  (in  Theme  124, Anomaly) 
r4.00,0.221MAINTENANCE  POLICIES  (in  Theme  176, Popularity) 
r4.00.0.041FQRMAL  REVIEW  (in  Theme  70, Anomaly) 
r4.00,0.061MANAGEMENT  REVIEWS  (in  Theme  15 7, Popularity) 
r4.00.0.031CRITICAL  AREAS  (in  Theme  1 04,Emerging) 
r4.00,0.131SUSTAINMENT  PLANNING  (in  Theme  15 7, Popularity) 
r4.00,0.031FUTURE  SYSTEMS  (in  Theme  1 76, Popularity) 
r4.00,0.021PERFQRMANCE  ISSUES  (in  Theme  46, Emerging) 
r4. 00.0.2 1 1UNFAVORABLE  COST  (in  Theme  1 76, Popularity) 
r4.00,0.171SUPPQRT  EFFORTS  (in  Theme  1 54, Emerging) 
r4.00,0.041SPECIFIC  TEST  (in  Theme  120, Emerging) 
r4.00,0.011SYSTEM  ANALYSIS  (in  Theme  15, Anomaly) 
r4.00.0.041EFFECTIVE  SYSTEMS  (in  Theme  99, Emerging) 
r4.00.0.751PROPQSED  TEST  (in  Theme  1 76, Popularity) 
r4.00,0.501QUALITY  PERFORMANCE  (in  Theme  46, Emerging) 
r4.00.0.041MANAGEMENT  CENTER  (in  Theme  169, Emerging) 
r4.00,0.021RELIABILITY  REQUIREMENTS  (in  Theme  153, Anomaly) 
r4.00.0.061MANAGEMENT  CRITERIA  (in  Theme  68, Anomaly) 
r4.00,0.501PRQCESS  BEGINS  (in  Theme  166, Anomaly) 
r4.00.0.041EFFECTlVE  TRAINING  (in  Theme  99, Emerging) 
r4.00,0.101INFORMATIQN  CPI  (in  Theme  141, Emerging) 
r4.00,0.0nSYSTEMS  DATA  (in  Theme  46, Emerging) 
r4.00,0.131SUPPQRT  CAPABILITIES  (in  Theme  67, Anomaly) 
r4.00.0.inASSESS  INFORMATION  (in  Theme  22, Anomaly) 
r4.00,0.0nSUSTAINMENT  PERFORMANCE  (in  Theme  157, Popularity) 
r4.00,0.671PRICE  INCENTIVE  (in  Theme  1 69, Emerging) 
r4. 00.0.0 llCRITICAL  PERFORMANCE  (in  Theme  68, Anomaly) 
r4.00,0.101USER  CAPABILITIES  (in  Theme  68, Anomaly) 
r4.00.0.061QPERAT10NAL  ASSESSMENTS  (in  Theme  172, Popularity) 
r4.00,0.1 11AFFORD ABILITY  COST  (in  Theme  1 54, Emerging) 
r4.00.0.1 11ACTUAL  OPERATIONAL  (in  Theme  99, Emerging) 
r4.00,0.041DEVELQPMENT  PROGRAMS  (in  Theme  120, Emerging) 
r4.00,0.041FUNCTIQNAL  CAPABILITY  (in  Theme  120, Emerging) 
r4.00,0.051INDEPENDENT  ASSESSMENT  (in  Theme  1 76, Popularity) 
r4.00,0.501CHANGE  ORDERS  (in  Theme  21, Anomaly) 
r4.00,0.171CQST  INFORMATION  (in  Theme  46, Emerging) 
r4.00,0.061INDIVIDUAL  EQUIPMENT  (in  Theme  106, Anomaly) 
r4.00,0.051LQGISTICS  COMPONENT  (in  Theme  15 7, Popularity) 
r4.00,0.101DQD  INFORMATION  (in  Theme  167, Popularity) 
r4.00,0.041LQGlSTlCS  SUPPORTABIL1TY  (in  Theme  108, Anomaly) 
r4.00,0.0210PERATIQNS  FACILITIES  (in  Theme  1 69, Emerging) 
r4.00,0.041SERVICE  MANAGEMENT  (in  Theme  15 7, Popularity) 
r4.00,0.071MISSIQN  SYSTEM  (in  Theme  172, Popularity) 
r4.00,0.041PRQDUCT  TECHNICAL  (in  Theme  1 69, Emerging) 
r4.00,0.031PRQDUCT  SUPPORT  PERFORMANCE  (in  Theme  149, Anomaly) 
r4.00,0.501RELATED  PROGRAM  (in  Theme  1 69, Emerging) 
[4.00,0.021MANAGEMENT  SERVICES  (in  Theme  179, Anomaly) 
r4.00,0.061DESIGN  DOCUMENTATION  (in  Theme  179, Anomaly) 
r4.00,1.001MAIS  ACQUISITION  (in  Theme  15 7, Popularity) 

[3.00, 1 .001DISTRIBUTIQN  SYSTEM  (in  Theme  1 72, Popularity) 
r3.00,0.031BUSlNESS  MANAGEMENT  (in  Theme  124, Anomaly) 
r3.00.0.061SPECIAL  SUPPORT  (in  Theme  1 69, Emerging) 
r3.00.0.141GQVERNMENT  SITE  (in  Theme  120, Emerging) 
r3.00, 1 .001PROVIDE  ADEQUATE  (in  Theme  1 76, Popularity) 
r3.00,0.031DQD  OVERSIGHT  (in  Theme  95, Anomaly) 
r3.00,0.021RELATED  MAINTENANCE  (in  Theme  1 69, Emerging) 
r3.00.0.021SUPPLY  SYSTEMS  (in  Theme  168, Anomaly) 
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f3 .00,0.01 1TRAINING  JOINT  (in  Theme  179, Anomaly) 
r3.00.0.021CQSTS  DESIGN  (in  Theme  120, Emerging) 
r3.00,0.131PRQGRAM  SCHEDULES  (in  Theme  176, Popularity) 
r3.00.0.031PRQGRAM  ELEMENTS  (in  Theme  1 1 1, Emerging) 
[3.00,0.061DEVELQPMENTAL  TESTS  (in  Theme  179, Anomaly) 
[3.00.0.141PBL  ARRANGEMENTS  (in  Theme  149, Anomaly) 
r3.00,0.401INFORMATIQN  SHARING  (in  Theme  46, Emerging) 
r3.00,0.021SYSTEMS  FACILITIES  (in  Theme  1 69, Emerging) 
r3.00,0.051CQMPUTER  PROGRAM  (in  Theme  99, Emerging) 
[3.00,0.02]OSDCOST  (in  Theme  166,  Anomaly) 
r3.00.0.091PRQVIDE  TIMELY  (in  Theme  155, Anomaly) 
r3.00,0.041DEVELQPMENT  CENTER  (in  Theme  1 69, Emerging) 
r3.00.0.071SUPPQRT  MODULE  (in  Theme  120, Emerging) 
[3.00,0.061DEFENSE  ADVANCED  (in  Theme  179, Anomaly) 
r3.00.0.061PROPQSED  PROGRAMS  (in  Theme  1 76, Popularity) 
r3 .00,0.041INFRASTRUCTURE  REQUIREMENTS  (in  Theme  99, Emerging) 
r3 .00.0.041 AIR  SYSTEM  (in  Theme  13, Anomaly) 

B.OO.O.OOIPROGRAM  SUPPORT  (in  Theme  1 76, Popularity) 
r3.00,0.031CQNTRACT  COST  (in  Theme  167, Popularity) 
r3.00,1.001PROTQTYPE  DEVELOPMENT  (in  Theme  124, Anomaly) 
r3 .00,0.01  PERSONNEL  AVAILABILITY  (in  Theme  106, Anomaly) 
r3.00,0.041CQNTRACT  REQUIREMENTS  (in  Theme  1 67, Popularity) 

[3 .00,0. 1 1 1COST  ANALYSES  (in  Theme  166, Anomaly) 
r3.00.0.081INCENTIVE  AWARD  (in  Theme  169, Emerging) 
r3.00,0.041CQNTRACT  REQUIRES  (in  Theme  167, Popularity) 
r3.00.0.021DEFENSE  CONTRACTORS  (in  Theme  108, Anomaly) 
r3.00,0.101IMPRQVED  MAINTENANCE  (in  Theme  46, Emerging) 
r3 .00.0.0 11DESIGN  TECHNICAL  (in  Theme  99, Emerging) 
r3.00,0.331TRAINING  PRODUCTS  (in  Theme  66, Anomaly) 
r3.00,0.081SUPPQRT  ASSETS  (in  Theme  91, Anomaly) 

[3 .00,0.0 1 1PRODUCT  SUPPORT  ACTIVITIES  (in  Theme  1 54, Emerging) 
r3.00.0.031REQUlREMENTS  DOCUMENTATION  (in  Theme  179, Anomaly) 
f3 .00,0. 1 1  PLANNING  ANALYSES  (in  Theme  166, Anomaly) 
r3.00,0.061PROPQSED  SUPPORT  (in  Theme  1 76, Popularity) 
r3.00,0.051PHASE  SYSTEM  (in  Theme  145, Anomaly) 
r3.00.0.071PRQGRAM  RISKS  (in  Theme  108, Anomaly) 
r3.00.0.021PERFQRMANCE  REPORTS  (in  Theme  168, Anomaly) 
r3.00,0.031REPQRTING  ANALYSIS  (in  Theme  157, Popularity) 
r3.00,0.021PERFQRMANCE  REPORTING  (in  Theme  157, Popularity) 
r3.00,0.031SUPPQRT  PROCUREMENT  (in  Theme  1 04, Emerging) 
r3.00.0.031PRQCESS  METRICS  (in  Theme  157, Popularity) 
r3 .00,0.031INFORMATIQN  CAPABILITIES  (in  Theme  67, Anomaly) 
r3.00.0.021ACQUISlT10N  PRODUCTION  (in  Theme  15 7, Popularity) 
r3.00,0.031SUSTAINMENT  PROCESSES  (in  Theme  99, Emerging) 

\3 .00,0.01 1MANAGEMENT  DESIGN  (in  Theme  99, Emerging) 
r3.00,0.081MEMQRANDUM  DATED  (in  Theme  145, Anomaly) 
r3.00.0.141SYSTEM  BASELINE  (in  Theme  1 76, Popularity) 
r3.00,0.061MEASURES  IDENTIFIED  (in  Theme  32, Anomaly) 

\3 .00,0.01 1LEVEL  PERFORMANCE  (in  Theme  106, Anomaly) 
r3.00.0.021ASSESSMENT  PROCESS  (in  Theme  15 7, Popularity) 
r3.00,0.071PRQCESS  DOCUMENTS  (in  Theme  56, Anomaly) 
r3. 00.0.01 1REOUIRED  SYSTEM  (in  Theme  67, Anomaly) 
r3.00,0.021CRITICAL  ITEMS  (in  Theme  68, Anomaly) 

[3.00,0. 17]DEFENSEDOD  (in Theme  167, Popularity) 
r3. 00,0.1 11ACTUAL  SYSTEM  (in  Theme  99, Emerging) 
r3.00,0.031TEST  REPORTING  (in  Theme  15 7, Popularity) 
r3.00,0.131RESQLVE  PROGRAM  (in  Theme  66, Anomaly) 

[3 .00,0.061 VULNERABILITY  PROGRAM  (in  Theme  70, Anomaly) 
r3.00,0.171DQD  FEDERAL  (in  Theme  1 54, Emerging) 
r3.00,0.021IDENTIFIED  TECHNICAL  (in  Theme  32, Anomaly) 
r3.00,0.041MANAGEMENT  OBJECTIVES  (in  Theme  1 76, Popularity) 
r3.00,0.131EQUIPMENT  SPARES  (in  Theme  106, Anomaly) 
r3. 00,0.01 1BUS1NESS  OPERATIONS  (in  Theme  1 69, Emerging) 
r3.00,0.171REPRESENTS  PROGRAM  (in  Theme  149, Anomaly) 
r3.00,0.501AVIAT10N  MAINTENANCE  (in  Theme  1 69, Emerging) 
r3.00,0.021CQST  ENGINEERING  (in  Theme  176,Popularity) 
r3.00,0.021MANPQWER  COST  (in  Theme  106, Anomaly) 
r3 .00,0.0 11TECHNICAL  INFORMATION  (in  Theme  46, Emerging) 
r3 .00,0.0 11RELIABILITY  SYSTEMS  (in  Theme  153, Anomaly) 
r3.00,0.031DEVELQPMENT  PROCUREMENT  (in  Theme  1 04, Emerging) 
r3.00,0.021RELATED  PERFORMANCE  (in  Theme  1 69, Emerging) 
r3.00,0.181SlTE  LOGISTICS  (in  Theme  120, Emerging) 
r3 .00,0.051PRQGRAM  RESPONSIBILITIES  (in  Theme  66, Anomaly) 
[3.00,0.1 11ASSESS  PERFORMANCE  (in  Theme  22, Anomaly) 
r3.00.0.061SYSTEM  TECHNICAL  (in  Theme  46, Emerging) 
r3.00.0.021E VALUATION  PERSONNEL  (in  Theme  106, Anomaly) 
r3.00, 1  .OOIFORCE  APPLICATION  (in  Theme  1 04, Emerging) 

[3.00,0. 501FINAL  DESIGN  (in  Theme  99, Emerging) 

[3 .00,0.03 1KEY  ACTIVITIES  (in  Theme  99, Emerging) 
r3.00.0.021SIGNIFICANT  COST  (in  Theme  149, Anomaly) 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-353- 


r3.00,0.041PRODUCTIQN  SUSTAINMENT  (in  Theme  1 57, Popularity) 
f3 .00.0.  Ill  APPLICATION  SYSTEM  (in  Theme  104, Emerging) 
r3.00,0.201DATA  SHARING  (in  Theme  46, Emerging) 
f3. 00.0.01 10PERAT10NS  PROGRAM  (in  Theme  1 69, Emerging) 
r3-00'0-051STANDARD  COMMERCIAL  (in  Theme  1 67, Popularity) 

[3. 00, 0.25]SERVICECOSI  (in  Theme  157, Popularity) 
r3.00,0.0411NTEGRATED  ELECTRONIC  (in  Theme  172, Popularity) 
r3.00,0.091EXECUTIVE  ORDER  (in  Theme  1 76, Popularity) 
r3.00.0.011TEST  SUPPORT  (in  Theme  1 76, Popularity) 
r3.00.0.071PRQCEDURES  BASED  (in  Theme  147, Anomaly) 
r3.00.0.021EFFECTIVE  LIFE  CYCLE  (in  Theme  99, Emerging) 
r3.00,0.021REPQRTING  SYSTEMS  (in  Theme  157,Popularity) 
f3 .00. 1.001 ADEQUATE  FUNDING  (in  Theme  1 76, Popularity) 
r3.00.0.061INTEROPERABILITY  REQUIREMENTS  (in  Theme  70, Anomaly) 
r3.00.0.131TECHNICAL  ISSUES  (in  Theme  46,Emerging) 
r3.00,0.041INDIVIDUAL  SYSTEM  (in  Theme  106, Anomaly) 
r3.00.0.021RELATED  COST  (in  Theme  1 69, Emerging) 
r3.00.0.021MAINTENANCE  EQUIPMENT  (in  Theme  1 69, Emerging) 

\3 .00.0.01 1MAINTENANCE  PROCESS  (in  Theme  157,Popularity) 
r3.00,0.021SPECIFlC  REQUIREMENTS  (in  Theme  120, Emerging) 
r3.00,0.151PRQCUREMENT  PROGRAM  (in  Theme  1 04, Emerging) 
r2.00.0.1410PERATIQNAL  CONTROLS  (in  Theme  32, Anomaly) 
r2.00.0.031DQD  REQUIREMENTS  (in  Theme  1 67, Popularity) 
f2 .00.0.0  UMETRICS  TIME  (in  Theme  15 7, Popularity) 
r2.00,0.031SYSTEM  PROVIDING  (in  Theme  99, Emerging) 
r2.00.0.031PM  MANAGEMENT  (in  Theme  147, Anomaly) 
r2.00.0.061FAILURES  TOTAL  (in  Theme  68, Anomaly) 
f2 .00.0.0  UEOUIPMENT  SOFTWARE  (in  Theme  169, Emerging) 
r2.00,0.021RELATED  TECHNICAL  (in  Theme  169, Emerging) 
r2.00, 1 .001CONTRACTQR  SELECTION  (in  Theme  42, Anomaly) 
r2.00.L001PERIQD  COVERED  (in  Theme  144, Anomaly) 
r2.00.0.041PRODUCTIQN  EFFORTS  (in  Theme  1 54, Emerging) 
r2.00,0.501DEMQNSTRATED  RESULTS  (in  Theme  52, Anomaly) 
[2.00,0.07]DLACAPDOD  (in  Theme  15, Anomaly) 
r2.00.0.021SEE  JOINT  (in  Theme  22, Anomaly) 
r2.00.0.031PRQGRAM  POLICY  (in  Theme  146, Anomaly) 
r2.00.0.041PERFQRM  DATA  (in  Theme  141, Emerging) 
r2.00,0.081QUANTITY  INITIAL  (in  Theme  52, Anomaly) 
r2.00.0.331PRQVlDE  DETAILS  (in  Theme  155, Anomaly) 
r2.00.0.141RCM  RELIABILITY  (in  Theme  153, Anomaly) 
r2.QO.O.OUPROCESS  DOCUMENT  (in  Theme  145, Anomaly) 
r2.00,0.021DATA  ACQUISITION  (in  Theme  15 7, Popularity) 
r2.00.0.011MISSIQN  PERFORMANCE  (in  Theme  46, Emerging) 
r2.00.0.331SUSTAINMENT  STRATEGY  (in  Theme  15 7, Popularity) 
[2.00,0.01]NAVYDATA  (in  Theme  124, Anomaly) 

[2.00,0.0 1]RISKSYSTEM  (in  Theme  1 76, Popularity) 
r2.00.0.031TOQLS  SUPPORT  (in  Theme  147, Anomaly) 
r2.00.0.021CQSTS  LOGISTICS  (in  Theme  120, Emerging) 
r2.00.0.011REQUIRED  CAPABILITY  (in  Theme  67, Anomaly) 
r2.00.0.041CQMPREHENSIVE  MAINTENANCE  (in  Theme  1 1 1, Emerging) 
r2.00,0.011SQFTWARE  SUPPORTABILITY  (in  Theme  169, Emerging) 
r2.00.0.011GQVERNMENT  FUNDING  (in  Theme  1 76, Popularity) 
f2. 00.0.03 1PROVIDE  SUPPORTING  (in  Theme  95, Anomaly) 
r2.00.0.011DESIGN  PROCESS  (in  Theme  157, Popularity) 
r2.00,0.021AREAS  MANAGEMENT  (in  Theme  1 04, Emerging) 
r2.00,0.031MULTIPLE  REQUIREMENTS  (in  Theme  1 67, Popularity) 
r2.00.0.061SOLUTIQNS  SUPPORT  (in  Theme  131, Anomaly) 
r2.00.0.031SUPPQRT  MULTIPLE  (in  Theme  1 67,Popularity) 
r2.00,0.011FUTURE  BUDGET  (in  Theme  1 76, Popularity) 
r2.00,0.031SYSTEM  FAILURES  (in  Theme  68, Anomaly) 

[2.00.0.031PARTS  BASED  (in  Theme  141, Emerging) 
r2.00.0.041DESIGN  SPECIFICATIONS  (in  Theme  108, Anomaly) 
r2.00,0.041MAlNTENANCE  FUNCTIONS  (in  Theme  1 1 1, Emerging) 
r2.00, 1 .001INTER  RELATED  (in  Theme  1 69, Emerging) 
r2.00.0.031PRQGRAM  ACTIONS  (in  Theme  104, Emerging) 
r2.00.0.021ACQUISITIQN  LOGISTICS  (in  Theme  15 7, Popularity) 
r2.00,0.201PRQGRAM  PROJECT  (in  Theme  42, Anomaly) 
r2.00.0.011DESIGN  INFORMATION  (in  Theme  46, Emerging) 
r2.00.0.071DET AILED  GUIDANCE  (in  Theme  1 76, Popularity) 
r2.00.0.021RELIABILITY  INFORMATION  (in  Theme  46, Emerging) 
r2.00.1.001STRATEGY  BEGINS  (in  Theme  166,Anomaly) 
r2.00,0.101RATES  PRODUCTION  (in  Theme  145, Anomaly) 
r2.00,0.031FACILITIES  DESIGN  (in  Theme  1 69, Emerging) 
r2.00.0.021MANAGEMENT  PROGRAMS  (in  Theme  1 20, Emerging) 
r2.00.0.031APPRQVED  PRIOR  (in  Theme  15 7, Popularity) 
r2.00.0.021SYSTEM  RELATED  (in  Theme  1 69, Emerging) 
r2.00.0.101PBL  AGREEMENTS  (in  Theme  153, Anomaly) 
[2.00.0.501PLANNING  EFFORTS  (in  Theme  120, Emerging) 
r2.00.0.011TECHNICAL  PROGRAM  (in  Theme  46, Emerging) 
r2.00.0.501REQUIREMENTS  SOURCE  (in  Theme  155, Anomaly) 
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[2.00,0. 171  SERVICE  REQUIREMENTS  (in  Theme  157, Popularity) 
r2.00.0.091TIMELY  INFORMATION  (in  Theme  155, Anomaly) 
[2.00,0.051STANDARD  EVALUATION  (in  Theme  1 67, Popularity) 
r2.00.0.011PRQGRAM  METRICS  (in  Theme  157, Popularity) 
[2.00,0.041FUNCTIQN  MISSION  (in  Theme  68, Anomaly) 
[2.00,0.101ANALYSIS  ACTIVITY  (in  Theme  1 54, Emerging) 
[2.00,0.071COMPQNENTS  REQUIRING  (in  Theme  106, Anomaly) 

[2.00,0.1 11REPAIRABLE  ASSETS  (in  Theme  91, Anomaly) 
12.00,0.011DEVELQPMENT  STRATEGY  (in  Theme  124, Anomaly) 
[2.00,0.501SUPPQRT  BEGINS  (in  Theme  166, Anomaly) 
r2.00.0.031ELEMENTS  MAINTENANCE  (in  Theme  1 1 1, Emerging) 

[2.00,0. 131FORMAL  RELIABILITY  (in  Theme  70, Anomaly) 

[2.00, 1 .001ADVANCED  DEVELOPMENT  (in  Theme  1 79, Anomaly) 
[2.00,0.001DESIGN  SERVICES  (in  Theme  99, Emerging) 
r2.00.0.011PERFQRMANCE  DESIGN  (in  Theme  46, Emerging) 
r2.00,1.001TIME  NEEDED  (in  Theme  141, Emerging) 
r2.00.0.021DESIGN  TEST  (in  Theme  1 76, Popularity) 

[2.00,0.0nSPECIFIC  PROGRAMS  (in  Theme  120, Emerging) 
r2.00,0.011ANALYSIS  MANAGEMENT  (in  Theme  15, Anomaly) 
r2.00,0.041SYSTEMS  REQUIRE  (in  Theme  46, Emerging) 
r2.00,0.031TECHNICAL  CAPABILITIES  (in  Theme  67,Anomaly) 
[2.00,0.501PRQGRAM  PRIOR  (in  Theme  15 7, Popularity) 
[2.00,0.021SPECIFIC  DESIGN  (in  Theme  120, Emerging) 
r2.00.0.131PRQGRAM  TYPE  (in  Theme  1 69, Emerging) 
r2.00,0.021ENSURE  READINESS  (in  Theme  172, Popularity) 
r2.00.0.081MAINTENANCE  PRACTICES  (in  Theme  131, Anomaly) 
[2.00,0.04HTEM  MANAGERS  (in  Theme  42, Anomaly) 
r2.00.0.011SYSTEMS  PERFORMANCE  (in  Theme  46, Emerging) 
[2.00,0.07]APBCOST  (in  Theme  15  7, Popularity) 

r2.00,0.041TRANSPORTATIQN  MANAGEMENT  (in  Theme  1 15, Anomaly) 
f2 .00,0.0 11INTEGRATED  TESTING  (in  Theme  179, Anomaly) 
r2.00.0.021TESTING  SUSTAINMENT  (in  Theme  179, Anomaly) 
r2.00,0.031PRlMARY  OBJECTIVES  (in  Theme  15 7, Popularity) 
r2.00,0.081QSD  LOGISTICS  (in  Theme  166, Anomaly) 

[2.00,0.101CPI  CRITICAL  (in  Theme  141, Emerging) 
r2.00.0.101TRAINING  CONCEPTS  (in  Theme  179, Anomaly) 
r2.00.0.051SEPARATE  PROGRAM  (in  Theme  15, Anomaly) 

[2.00, 1 .001SYSTEM  MODIFICATION  (in  Theme  46, Emerging) 
r2.00.0.501IDENTIFY  REQUIRED  (in  Theme  67, Anomaly) 
r2.00,1.001DESIGN  STANDARD  (in  Theme  167, Popularity) 
r2.00.1  .OOIMAJOR  FEDERAL  (in  Theme  1 54,Emerging) 
r2.00,0.021INTEGRATIQN  RISK  (in  Theme  1 76, Popularity) 
r2.00.0.011TECHNICAL  REPORTS  (in  Theme  168, Anomaly) 
[2.00,0.03TTERM  PROGRAM  (in  Theme  108, Anomaly) 

[2.00,0.201ASSESS  PROGRAM  (in  Theme  22, Anomaly) 
r2.00,0.02HDENTIFY  RISK  (in  Theme  67, Anomaly) 
r2.00.0.011EVALUATIQN  CENTER  (in  Theme  1 69, Emerging) 
f2 .00,0.0 11SUPPORT  PROCESS  (in  Theme  15 7, Popularity) 
r2.00,0.011EVALUATIQN  PROCESS  (in  Theme  15 7, Popularity) 
f2 .00.0.251 APPROVED  CONTRACT  (in  Theme  75, Anomaly) 
r2.00,0.041FQRMAL  TESTING  (in  Theme  70, Anomaly) 
r2.00,0.061CQST  CONSIDERATIONS  (in  Theme  1 54, Emerging) 
[2.00,0.031SYSTEM  INTERFACE  (in  Theme  99, Emerging) 
r2.00.0.031PERFQRMANCE  RISKS  (in  Theme  108,Anomaly) 
r2.00,0.051BUDGET  DATA  (in  Theme  1 76, Popularity) 
r2.00,0.501SUBMlT  PROGRAM  (in  Theme  1 76, Popularity) 
r2.00,0.031CQNDUCT  MAINTENANCE  (in  Theme  46, Emerging) 
r2.00,0.031PRIMARY  SUSTAINMENT  (in  Theme  15 7, Popularity) 
r2.00,0.051SUPPQRTING  SYSTEM  (in  Theme  95, Anomaly) 
r2.00,0.021REVIEWS  PERFORMANCE  (in  Theme  157,Popularity) 
r2.00,0.091DISPQSAL  PLANNING  (in  Theme  120, Emerging) 
r2.00,0.071SYSTEMS  AGENCY  (in  Theme  1 54, Emerging) 
r2.00,0.021SINGLE  CONTRACT  (in  Theme  145, Anomaly) 
r2.00,0.011INITIAL  OPERATING  (in  Theme  1 76, Popularity) 
[2.00,0.0nTESTlNG  REQUIRED  (in  Theme  179, Anomaly) 
r2.00,0.031PRQGRAM  FUNCTIONS  (in  Theme  1 1 1, Emerging) 
[2.00,0.01INAVY  PROGRAM  (in  Theme  124, Anomaly) 
r2.00,0.071MILITARY  PERSONNEL  (in  Theme  167, Popularity) 
r2.00,0.031REQUIRE  SUPPORT  (in  Theme  46, Emerging) 
[2.00,0.0nMANAGEMENT  ISSUES  (in  Theme  46, Emerging) 
r2.00,0.201INTERFACE  CONTROL  (in  Theme  172, Popularity) 
r2.00,0.051EQUIPMENT  COMPONENT  (in  Theme  15 7, Popularity) 
r2.00,0.001BASED  PROCESS  (in  Theme  157, Popularity) 
r2.00,0.091ENGlNEERING  CONCERNS  (in  Theme  124, Anomaly) 
r2.00.1.001GUIDES  LIFE  CYCLE  (in  Theme  108, Anomaly) 
r2.00.0.041REQUlRES  ACQUISITION  (in  Theme  1 67, Popularity) 
r2.00,0.021DETAILED  ENGINEERING  (in  Theme  1 76, Popularity) 
r2.00,0.061CRlTERIA  EQUIPMENT  (in  Theme  68, Anomaly) 
f2 .00,0.0 11REQUIRED  REPORTS  (in  Theme  168, Anomaly) 
r2.00.0.081FUNDING  CONTRACT  (in  Theme  1 76, Popularity) 
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r2.00,0.061PHYSICAL  CAPABILITIES  (in  Theme  67, Anomaly) 
r2.00.0.101ENSURE  TIMELY  (in  Theme  155, Anomaly) 
r2.00,0.061WORKLQAD  ALLOCATION  (in  Theme  68, Anomaly) 
r2.00.0.031INTEGRATIQN  PROGRAM  (in  Theme  42, Anomaly) 
r2.00,0.041MANAGEMENT  FUNCTIONS  (in  Theme  1 1 1, Emerging) 
r2.00,1.001DETAILED  PROGRAM  (in  Theme  1 76, Popularity) 
r2.00,0.021INFRASTRUCTURE  MAINTENANCE  (in  Theme  99, Emerging) 
r2.00,0.011TIME  COST  (in  Theme  141, Emerging) 
r2.00,0.021FUTURE  MAINTENANCE  (in  Theme  1 76, Popularity) 
r2.00,0.171DATA  ENTRY  (in  Theme  95, Anomaly) 
r2.00.0.041QRGANIC  SOFTWARE  (in  Theme  169,Emerging) 
r  1.00,0.0 11ACQUISITION  PLANNING  (in  Theme  15 7, Popularity) 
ri.OO.O.OOlDATA  SYSTEMS  (in  Theme  1 72, Popularity) 
r  1 .00,0.03 1FACILITIES  HARDWARE  (in  Theme  169, Emerging) 
r  1.00.0.01 1DEVELOPMENT  RELIABILITY  (in  Theme  124, Anomaly) 
ri  .00,0.0 11PBL  APPROACH  (in  Theme  67, Anomaly) 
ri.00.0.031DQD  PROGRAMS  (in  Theme  167, Popularity) 
ri .00,0.0 11TEST  PLANNING  (in  Theme  1 76, Popularity) 

[  1.00, 0.03 ]DODRAM  (in  Theme  167, Popularity) 
ri .00,0.0 11TECHNICAL  ASSESSMENT  (in  Theme  46, Emerging) 
ri .00,0.0 11DESIGN  ACQUISITION  (in  Theme  157,Popularity) 
ri.00,0.031PRQDUCT  SUPPORT  STRATEGY  (in  Theme  149, Anomaly) 
ri.00,0.021DQD  MODELING  (in  Theme  1 67, Popularity) 
ri .00.0.0 llSUPPORT  ACQUISITION  (in  Theme  157,Popularity) 
ri.OO.O.OOlSYSTEMS  ACQUISITION  (in  Theme  15 7, Popularity) 
r  1 .00.0. 1 31CONTRACT  TYPE  (in  Theme  1 67, Popularity) 
ri  .00.0.01 1LOGISTICS  TRAINING  (in  Theme  1 69, Emerging) 
ri.OO.O.OOlCONTROL  ACTIVITIES  (in  Theme  172, Popularity) 
ri.00,0.021SPECIFIC  FAILURE  (in  Theme  120, Emerging) 
fl  .00,0.0 11PERFORMANCE  ASSESSMENT  (in  Theme  46, Emerging) 
n  .00,0.0 11SERVICE  CONTRACTS  (in  Theme  15 7, Popularity) 
ri.OO.O.lOlPROCUREMENT  PROCESS  (in  Theme  15 7, Popularity) 
ri.00,0.021SYSTEM  PERFORMANCE  (in  Theme  172, Popularity) 
ri.00,0.031DQD  ORGANIZATIONS  (in  Theme  167, Popularity) 
r  1.00,0.01  PERFORMANCE  ENGINEERING  (in  Theme  46, Emerging) 
r  1  .OO.O.Q41PRODUCTION  DEPLOYMENT  (in  Theme  157, Popularity) 
r  1 .00.0.021MATERIAL  COSTS  (in  Theme  120, Emerging) 
ri.00,0.0nSYSTEM  PROGRAM  (in  Theme  1 76, Popularity) 
r  1 .00.0.031QPERAT10NS  SUSTAINMENT  (in  Theme  15 7, Popularity) 
ri.00,0.001PRQCUREMENT  PACKAGE  (in  Theme  120, Emerging) 


Appendix  B:  Overview  of  Lexical  Link  Analysis 

As  in  military  operations,  where  the  term  situational  awareness  was  coined,  we  note 
that  our  efforts  can  inform  awareness  of  analyzed  data  in  a  unique  way  that  helps  improve  a 
decision-maker’s  understanding  or  awareness  of  the  data’s  content.  We,  therefore,  define 
awareness  as  the  cognitive  interface  between  decision-makers  and  a  complex  system, 
expressed  in  a  range  of  terms  or  features,  or  a  specific  vocabulary  or  lexicon,  to  describe 
the  attributes  and  surrounding  environment  of  the  system.  Specifically,  LLA  is  a  form  of  text 
mining  in  which  word  meanings  represented  in  lexical  terms  (e.g.,  word  pairs)  can  be 
represented  as  if  they  are  in  a  community  of  a  word  network. 

Link  analysis  discovers  and  displays  a  network  of  word  pairs.  These  word  pair 
networks  are  characterized  by  one-,  two-,  or  three-word  themes.  Figure  6  shows  a 
visualization  of  common  lexical  links  shared  between  Systems  1  and  2,  shown  in  the  red 
box.  A  system,  or  a  corpus,  can  be  a  collection  of  documents  for  an  actual  physical  system 
(e.g.,  acquisition  strategies  for  a  Navy  ship-building  program)  or  simply  a  category  of 
information.  A  node  in  Figure  6  represents  a  word  in  a  corpus  and  a  link  or  edge  represents 
a  word  pair.  A  word  pair  is  a  bi-gram  (Manning  &  Schutze,  1999)  word  pair  extracted  from 
the  corpus.  Within  the  field  of  computational  linguistics,  an  n-gram  is  a  sequence  of  n  items 
matched  to  certain  probabilistic  patterns  from  a  given  text.  Size  2  of  an  n  -gram  is  a  bi-gram. 
In  Figure  6,  each  link  color  refers  to  the  collection  of  words,  lexicon,  or  features  that  belongs 
to  a  cluster  that  describes  a  concept  or  theme.  In  overlapping  areas,  nodes  are  lexically 
linked.  Unlinked,  outer  vectors  (outside  the  red  box)  indicate  unique  system  features.  Figure 
7  shows  the  information  from  three  categories  that  can  be  compared,  and  Figure  8  shows 
the  information  from  two  time  periods  that  can  be  compared.  What  is  unique  here  is  that  LLA 
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constructs  these  linkages  via  intelligent  agent  technology  using  social  network  grouping 
methods. 

The  closeness  of  the  systems  in  comparison  can  be  examined  visually  or  using  the 
quadratic  assignment  procedure  (QAP;  Hubert  &  Schultz,  1976  [e.g.,  in  UCINET];  Borgatti, 
Everett,  &  Freeman,  2002)  to  compute  the  correlation  of  two  sets  of  lexical  terms  from  two 
systems  and  analyze  the  structural  differences  in  the  two  systems,  as  shown  in  Figure  9. 


Figure  6.  Comparing  Two  Systems  Using  LLA 


Figure  7.  Comparing  Three  Categories 
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Figure  8.  Comparing  Two  Time  Periods 
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Figure  9.  QAP  Correlation  via  UCINET 

Figure  10  shows  a  visualization  of  LLA  with  connected  keywords  or  concepts  as 
clusters,  groups,  or  themes.  Words  are  linked  as  word  pairs  that  appear  next  to  each  other 
in  the  original  documents.  Different  colors  indicate  different  clusters  of  word  groups.  They 
were  produced  using  a  social  network  community  detection  method  (Girvan  &  Newman, 
2002)  where  words  are  connected,  as  shown  in  a  single  color,  as  if  they  are  in  a  social 
community.  A  word  center  is  formed  around  a  word  node  connected  with  a  list  of  other 
words  in  word  pairs.  For  instance,  Figure  1 1  shows  a  detailed  view  of  a  theme  or  word  group 
in  Figure  10.  The  center  words  are  analysis,  research,  and  approach.  In  this  example,  we 
use  three  words  such  as  “analysis,  research,  approach”  to  label  such  a  group,  where  the  top 
three  words  are  those  with  the  highest  total  degree  of  centralities  (Freeman,  1979; 
Wasserman  &  Faust,  1994). 
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Figure  10.  Word  and  Term  of  Themes  Discovered  and  Shown  in  Colored  Groups 


Figure  11.  A  Detailed  View  of  a  Theme  or  Word  Group  From  Figure  10 

The  detailed  steps  of  LLA  processing  include  the  following: 

•  Step  1 :  Select  word  pairs  based  on  the  following  bi-gram  parameters: 

o  the  probability  threshold  for  one  word  next  to  another  word  in  a  word 
pair  and 
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o  the  minimum  frequency  for  each  individual  word. 

•  Step  2:  Apply  a  social  network  community-finding  algorithm  (i.e.,  Newman 
community  detection  method;  Girvan  &  Newman,  2002)  to  group  the  word 
pairs  into  themes.  A  theme  includes  a  cluster  of  lexical  word  pairs  connected 
to  each  other. 

•  Step  3:  Compute  a  “weight,”  or  an  importance  measure,  for  a  theme. 

•  Step  4:  Sort  theme  weights  by  time  and  study  the  distributions  of  the  themes 
by  time. 

The  outputs  of  LLA  include  lexical  network  visualizations  such  as  the  ones  in  Figures 
6-11;  radar  visualization;  and  matrix  visualization  (Zhao,  Gallup,  &  MacKinnon,  2010).  The 
word  pair  groups  or  themes  as  shown  Figure  10  and  1 1  are  further  divided  into  three  types 
according  to  the  weights  in  Step  3: 

•  Popular  (P):  themes  containing  the  highest  number  of  mutually  connected 
word  pairs.  The  themes  represent  the  main  topics  in  a  corpus  at  the  time.  The 
theme  represented  in  Figure  11  is  an  example  of  a  popular  theme. 

•  Emerging  (E):  themes  containing  the  medium  number  of  mutually  connected 
word  pairs.  These  themes  may  grow  to  be  popular  over  time. 

•  Anomalous  (A):  themes  containing  the  lowest  number  of  mutually  connected 
word  pairs.  These  themes  may  be  off-topics  compared  to  other  themes  and 
may  be  interesting  for  further  investigation. 

Business  Problems  That  LLA  Addresses 

As  a  text  analysis  tool,  LLA  typically  addresses  the  business  problems  of  discovering 
themes  and  topics  in  unstructured  documents  and  sorting  the  importance  of  the  themes 
accordingly.  Current  methods,  for  example,  internet  search  methods  of  ranking  pages, 
require  established  hyperlinks,  citation  networks,  or  other  forms  of  crowd-sourced  collective 
intelligence.  LLA  is  especially  useful  for  data  without  hyperlinks  and  citation  networks,  for 
example,  large-scale  government  internal  documents.  Furthermore,  current  methods 
typically  rank  the  importance  of  the  information  based  on  its  popularity.  For  example,  we 
found  that  in  many  business  applications,  it  is  useful  to  rank  information  based  on  emerging 
importance  or  anomalousness. 

Current  research  on  social  network  analysis  focuses  mostly  on  people  or 
organizations  with  direct  associations  regardless  of  the  contents  linked.  The  so-called  study 
of  centrality  (Girvan  &  Newman,  2002;  Feldman  &  Sanger,  2007)  has  been  a  focal  point  for 
the  social  network  structure  study.  Finding  the  centrality  of  a  network  lends  insight  into  the 
various  roles  and  groupings  such  as  the  connectors  (e.g.,  mavens,  leaders,  bridges,  isolated 
nodes),  the  clusters  (and  who  is  in  them),  the  network  core,  and  its  periphery  (Orgnet, 

2011). 

One  of  the  core  innovations  of  LLA  is  to  analyze  the  content  (e.g.,  documents  and 
social  media  communications)  created  by  social  entities  (e.g.,  people  or  organizations)  and, 
therefore,  create  alternative  networks  (i.e.,  semantic  networks)  to  traditional  social  networks. 
The  resulting  networks  from  LLA  examine  both  social  and  semantic  networks  in  terms  of  the 
organizations  and  people  involved  in  the  important  themes,  and  how  semantic  networks 
might  suggest  improved  potential  collaborations  and  predict  future  outcomes. 
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Implementation  Details 

In  the  past  year,  we  continued  our  efforts  at  the  Naval  Postgraduate  School  (NPS)  by 
using  collaborative  learning  agents  (CLAs;  Ql,  2009)  and  other  tools,  including  AutoMap 
(Center  for  Computational  Analysis  of  Social  and  Organizational  Systems  [CASOS],  2009) 
for  improved  visualizations.  Results  from  these  efforts  arose  from  leveraging  intelligent 
agent  technology  via  an  educational  license  with  Quantum  Intelligence,  Inc.  CLA  is  a 
computer-based  learning  agent,  or  agent  collaboration  tool,  capable  of  ingesting  and 
processing  data  sources. 

We  have  been  generating  visualizations  including  a  lexical  network  visualization 
using  various  open  source  tools.  We  began  by  using  the  Organizational  Risk  Assessment 
(ORA;  CASOS,  2009)  tool  and  expanded  to  other  tools.  For  example,  in  the  past  year,  we 
developed  3-D  network  views  using  Pajek  (Batagelj,  Mrvar  &  Zaversnik,  2011)  and  X3D 
(Web3D,  201 1 ).  We  also  developed  our  visualizations  radar  view  and  match  view  (Zhao  at 
al. ,  2010). 

LLA  uses  a  computer-based  learning  agent  called  CLA  (Ql,  2009)  to  employ  an 
unsupervised  learning  process  that  separates  patterns  and  anomalies.  Unsupervised  agent 
learning  is  implemented  by  indexing  each  set  of  documents  separately  and  in  parallel  using 
multiple  learning  agents.  Unsupervised  agents  are  used  because  the  learning  data  for 
supervised  agents  are  expensive  to  obtain.  Multiple  agents  can  work  collaboratively  and  in 
parallel.  We  set  up  a  cluster  utilizing  Linux  servers  in  the  NPS  High  Performance  Computing 
Center  (HPC)  to  handle  the  large-scale  data  and  the  secure  environment  in  the  NPS  Secure 
Technology  Battle  Laboratory  (STBL). 

Relations  to  Other  Methods 

The  LLA  approach  is  more  properly  related  to  latent  semantic  analysis  (LSA; 

Dumais,  Furnas,  Landauer,  &  Deerwester,  1988)  and  probabilistic  latent  semantic  analysis 
(PLSA).  In  the  LSA  approach,  a  term-document  matrix  is  the  starting  point  for  analysis.  The 
elements  of  the  term-document  or  feature-object  (term  as  feature  and  document  as  object) 
matrix  are  the  occurrences  of  each  word  in  a  particular  document  (i.e.,  A  =  a^j,  where  atj 
denotes  the  frequency  in  which  term  j  occurs  in  document  i).  The  term-document  matrix  is 
usually  sparse.  LSA  uses  singular  value  decomposition  (SVD)  to  reduce  the  dimensionality 
of  the  term-document  matrix.  SVD  cannot  be  applied  to  the  cases  where  the  vocabulary  (the 
unique  number  of  terms)  in  the  document  collection  is  large;  for  example,  the  number  of 
unique  terms  in  the  DoD’s  acquisition  documentation  approaches  the  “large”  value  that 
would  make  SVD  inapplicable.  LSA  has  been  widely  used  to  improve  information  indexing, 
search/retrieval,  and  text  categorization. 

A  recent  development  related  to  this  method  is  called  latent  Dirichlet  allocation  (LDA; 
Blei,  Ng,  &  Jordan,  2003),  which  is  a  generative  probabilistic  model  of  a  corpus.  In  LDA,  a 
document  is  considered  to  be  composed  of  a  collection  of  words,  a  “bag  of  words,”  where 
word  order  and  grammar  are  not  considered  important.  The  basic  idea  is  that  documents 
are  represented  as  random  mixtures  over  latent  topics,  where  each  topic  is  characterized  by 
a  statistical  distribution  (Dirichlet  distribution)  over  the  corpus. 

Our  theme  generation  from  LLA  is  different  than  from  LDA,  in  which  a  collection  of 
lexical  terms  is  connected  to  each  other  semantically,  as  if  the  terms  are  in  a  social 
community,  and  social  network  grouping  methods  are  used  to  group  the  words.  Also  unlike 
LSA,  our  method  is  easily  scaled  to  analyze  a  large  vocabulary  and  is  generalizable  to  any 
sequential  data. 
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Anticipated  Benefits 

Our  LLA  method  provides  solutions  to  meet  the  critical  needs  of  the  acquisition 
research  community.  The  key  advantage  is  to  provide  an  innovative  near-real-time  self- 
awareness  system  to  transfer  diversified  data  services  into  strategic  decision-making 
knowledge,  specifically  through  the  following: 

•  Automation:  High  correlation  of  LLA  results — with  the  link  analysis  done  by 
human  analysts — makes  it  possible  to  save  human  power  and  improve 
responsiveness.  Automation  is  achieved  via  computer  program  or  software 
agents  to  perform  LLA  frequently — and  in  near  real-time. 

•  Discovery:  LLA  “discovers”  and  displays  a  network  of  word  pairs.  These  word 
pair  networks  are  characterized  by  one-,  two-,  or  three-word  themes.  The 
weight  of  each  theme  is  determined  based  on  its  frequency  of  occurrence. 
LLA  may  also  discover  blind  spots  of  human  analysis  that  are  caused  by  the 
overwhelming  amount  of  data  for  human  analysts  to  consider. 

•  Validation:  LLA  may  provide  different  perspectives  of  links.  In  the  acquisition 
context,  links  discovered  by  human  analysts  may  emphasize  component  and 
part  connections  that  do  not  necessarily  reflect  content 

overlaps. Consequently,  it  can  provide  improved  results  in  terms  of  trust, 
quality  of  association,  and  discovery;  can  help  break  through  the  taxonomy  of 
ignorance  (Denby  &  Gammack,  1999)  and  organizational  boundaries;  and 
can  help  improve  organizational  reach. 
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